Skip to content

CLDSRV-910: Rewrite the data of a self-copy when its encryption changes - #6346

Merged
bert-e merged 6 commits into
development/9.3from
bugfix/CLDSRV-910-reencrypt-data-on-self-copy
Nov 4, 2026
Merged

bert-e merged 6 commits into
development/9.3from
bugfix/CLDSRV-910-reencrypt-data-on-self-copy

Conversation

@dvasilas

Copy link
Copy Markdown
Contributor

A CopyObject of an object onto itself (same bucket and key, same location, versioning disabled) didn't rewrite the data, even when the copy changed the encryption.

This is the recommended way to encrypt an unencrypted object or to re-encrypt under another master key

  • Using it with an unencrypted object left the data in plaintext while the metadata and responses said it was encrypted.
  • Using it to re-encrypt recorded the new key in the metadata but kept the data key wrapped under the old one. So GETs decrypted with the wrong data key and returned wrong data.

With this PR, a self-copy rewrites the data when the destination is encrypted, and the source data is either unencrypted or encrypted under another master key. A self-copy under the same master key stays metadata-only.

The PR cherry-picks CLDSRV-908, so that a self-copy that rewrites the data deletes the old data.

Note: lib/kms/wrapper.js is reformatted with prettier in a separate commit.

@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Hello dvasilas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-910 contains:

  • 9.3.26

  • 9.4.11

  • 9.6.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.3.26

  • 9.4.11

  • 9.5.3

  • 9.6.0

Please check the Fix Version/s of CLDSRV-910, or the target
branch of this pull request.

Comment thread lib/kms/wrapper.js Dismissed
Comment thread lib/kms/wrapper.js Dismissed
Comment thread lib/kms/wrapper.js Dismissed
Comment thread lib/kms/wrapper.js Dismissed
Comment thread lib/kms/wrapper.js Dismissed
Comment thread lib/kms/wrapper.js Dismissed
@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

@dvasilas

Copy link
Copy Markdown
Contributor Author

/create_integration_branches

@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.61905% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.45%. Comparing base (649530a) to head (681a0ae).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
lib/kms/wrapper.js 65.15% 23 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
lib/api/objectCopy.js 92.53% <100.00%> (+1.19%) ⬆️
lib/kms/wrapper.js 76.38% <65.15%> (+1.98%) ⬆️

... and 1 file with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.3    #6346      +/-   ##
===================================================
+ Coverage            85.35%   85.45%   +0.09%     
===================================================
  Files                  207      207              
  Lines                13468    13487      +19     
===================================================
+ Hits                 11496    11525      +29     
+ Misses                1972     1962      -10     
Flag Coverage Δ
file-ft-tests 68.48% <69.04%> (+0.14%) ⬆️
file-ft-tests-null-compat 68.94% <69.04%> (+0.14%) ⬆️
kmip-ft-tests 28.37% <36.90%> (+0.02%) ⬆️
mongo-v0-ft-tests 69.64% <69.04%> (+0.14%) ⬆️
mongo-v1-ft-tests 69.69% <69.04%> (+0.18%) ⬆️
multiple-backend 36.83% <46.42%> (-0.03%) ⬇️
s3c-ft-tests-v0 64.07% <69.04%> (+0.04%) ⬆️
s3c-ft-tests-v0-null-compat 64.17% <69.04%> (+0.08%) ⬆️
s3c-ft-tests-v1 64.05% <69.04%> (+0.04%) ⬆️
sur-tests 35.90% <3.57%> (-1.01%) ⬇️
sur-tests-inflights 37.73% <3.57%> (-0.07%) ⬇️
unit 71.95% <72.61%> (+0.18%) ⬆️
utapi-v2-tests 34.65% <15.47%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread lib/api/objectCopy.js Resolved
@dvasilas
dvasilas force-pushed the bugfix/CLDSRV-910-reencrypt-data-on-self-copy branch from 19797ca to f454274 Compare October 28, 2026 15:17
@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

History mismatch

Merge commit #b754602cdec1f8112d03bb5a3d69865f654808ce on the integration branch
w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy is merging a branch which is neither the current
branch bugfix/CLDSRV-910-reencrypt-data-on-self-copy nor the development branch
development/9.4.

It is likely due to a rebase of the branch bugfix/CLDSRV-910-reencrypt-data-on-self-copy and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

@dvasilas

Copy link
Copy Markdown
Contributor Author

/force_reset

@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

@scality scality deleted a comment from bert-e Oct 28, 2026
@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Conflict

A conflict has been raised during the creation of
integration branch w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy with contents from bugfix/CLDSRV-910-reencrypt-data-on-self-copy
and development/9.4.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy origin/development/9.4
 git merge origin/bugfix/CLDSRV-910-reencrypt-data-on-self-copy
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy

The following options are set: create_integration_branches

@scality scality deleted a comment from bert-e Oct 28, 2026
@scality scality deleted a comment from bert-e Oct 28, 2026
@scality scality deleted a comment from bert-e Oct 28, 2026
@scality scality deleted a comment from bert-e Oct 28, 2026
@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Conflict

A conflict has been raised during the creation of
integration branch w/9.5/bugfix/CLDSRV-910-reencrypt-data-on-self-copy with contents from w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy
and development/9.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.5/bugfix/CLDSRV-910-reencrypt-data-on-self-copy origin/development/9.5
 git merge origin/w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.5/bugfix/CLDSRV-910-reencrypt-data-on-self-copy

The following options are set: create_integration_branches

@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1
  • development/9.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

@bert-e

bert-e commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

@dvasilas
dvasilas requested a review from leif-scality October 30, 2026 08:02
Comment thread lib/api/objectCopy.js Outdated
Comment on lines +79 to +80
!location.masterKeyId ||
!masterKeyId ||

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why do we need to reencrypt if neither the existing encryption nor the configured encryption have a master key ID?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. This checks two cases that normally cannot happen:

  • !location.masterKeyId: The source object is encrypted (location.cipheredDataKey), but the metadata is missing the master key ID, so we can't tell whether it is the same key.
  • !masterKeyId: The copy asks for encryption, but the destination has no master key ID

They cover potential regressions, we could arguably drop them to simplify the code.

But it was confusing to readers. I added a new commit to use kms.isSameMasterKey instead (returns false when either key is missing) and added the case to the function doc.

@bert-e

bert-e commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Conflict

There is a conflict between your branch bugfix/CLDSRV-910-reencrypt-data-on-self-copy and the
destination branch development/9.3.

Please resolve the conflict on the feature branch (bugfix/CLDSRV-910-reencrypt-data-on-self-copy).

git fetch && \
git checkout origin/bugfix/CLDSRV-910-reencrypt-data-on-self-copy && \
git merge origin/development/9.3

Resolve merge conflicts and commit

git push origin HEAD:bugfix/CLDSRV-910-reencrypt-data-on-self-copy

The following options are set: create_integration_branches

@dvasilas
dvasilas force-pushed the bugfix/CLDSRV-910-reencrypt-data-on-self-copy branch from e41b544 to 681a0ae Compare November 3, 2026 17:39
@bert-e

bert-e commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

History mismatch

Merge commit #ba2f7f3b1fa90db14b23a6bad738b2db285f4ae1 on the integration branch
w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy is merging a branch which is neither the current
branch bugfix/CLDSRV-910-reencrypt-data-on-self-copy nor the development branch
development/9.4.

It is likely due to a rebase of the branch bugfix/CLDSRV-910-reencrypt-data-on-self-copy and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

Comment thread lib/api/objectCopy.js Resolved
@dvasilas

dvasilas commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/force_reset

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Conflict

A conflict has been raised during the creation of
integration branch w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy with contents from bugfix/CLDSRV-910-reencrypt-data-on-self-copy
and development/9.4.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy origin/development/9.4
 git merge origin/bugfix/CLDSRV-910-reencrypt-data-on-self-copy
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy

The following options are set: create_integration_branches

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Conflict

A conflict has been raised during the creation of
integration branch w/9.5/bugfix/CLDSRV-910-reencrypt-data-on-self-copy with contents from w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy
and development/9.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.5/bugfix/CLDSRV-910-reencrypt-data-on-self-copy origin/development/9.5
 git merge origin/w/9.4/bugfix/CLDSRV-910-reencrypt-data-on-self-copy
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.5/bugfix/CLDSRV-910-reencrypt-data-on-self-copy

The following options are set: create_integration_branches

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

@dvasilas

dvasilas commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/approve

@dvasilas

dvasilas commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/status

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Status

check status
➡️ Approvals ☀️
➡️ Integration builds ☀️
➡️ Fix versions ☀️
➡️ History ❗ reset may be needed

The following options are set: approve, create_integration_branches

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Queue out of order

The changeset has received all authorizations to enter the merge queue,
however the merge queue is currently in an incoherent state. In order
to protect this pull request, I have not added the changeset to the
queue.

Look into https://devdocs.scality.net/support/faq/#bert-e for guidance about
what to do next to unlock the situation, or ask for help on the #ci-support
Slack channel.

The following options are set: approve, create_integration_branches

@dvasilas

dvasilas commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/status

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Status

check status
➡️ Approvals ☀️
➡️ Integration builds ☀️
➡️ Fix versions ☀️
➡️ History ❗ reset may be needed

The following options are set: approve, create_integration_branches

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/9.3

  • ✔️ development/9.4

  • ✔️ development/9.5

  • ✔️ development/9.6

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1
  • development/9.2

This pull request does not target the following hotfix branch(es) so they
will be left untouched:

  • hotfix/7.10.8
  • hotfix/7.70.73
  • hotfix/7.10.30
  • hotfix/7.10.2
  • hotfix/7.8.0
  • hotfix/7.10.28
  • hotfix/7.4.9
  • hotfix/7.4.0
  • hotfix/7.10.3
  • hotfix/7.7.0
  • hotfix/7.10.49
  • hotfix/7.70.11
  • hotfix/7.9.0
  • hotfix/7.10.15
  • hotfix/9.3.13
  • hotfix/7.4.5
  • hotfix/7.70.51
  • hotfix/7.4.4
  • hotfix/7.6.0
  • hotfix/8.8.45
  • hotfix/9.2.24
  • hotfix/7.10.27
  • hotfix/7.4.1
  • hotfix/7.10.1
  • hotfix/7.4.6
  • hotfix/7.4.8
  • hotfix/7.2.0
  • hotfix/9.2.36
  • hotfix/7.10.0
  • hotfix/7.4.10
  • hotfix/9.0.32
  • hotfix/7.4.2
  • hotfix/7.70.45
  • hotfix/9.0.7
  • hotfix/7.4.7
  • hotfix/7.70.21
  • hotfix/6.4.7
  • hotfix/7.4.3
  • hotfix/7.10.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

@bert-e

bert-e commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.3

  • ✔️ development/9.4

  • ✔️ development/9.5

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1
  • development/9.2

Please check the status of the associated issue CLDSRV-910.

Goodbye dvasilas.

@bert-e
bert-e merged commit 010b644 into development/9.3 Nov 4, 2026
35 of 37 checks passed
@bert-e
bert-e deleted the bugfix/CLDSRV-910-reencrypt-data-on-self-copy branch November 4, 2026 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants