Skip to content

HDDS-16677. Support create MPU key with StoragePolicy - #11626

Merged
devmadhuu merged 10 commits into
apache:HDDS-11233from
anuragp010:HDDS-16677
Nov 17, 2026
Merged

devmadhuu merged 10 commits into
apache:HDDS-11233from
anuragp010:HDDS-16677

Conversation

@anuragp010

@anuragp010 anuragp010 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Co-authored-by: https://github.com/xichen01

What changes were proposed in this pull request?

MPU Keys do not support Storage Policy. This PR adds support for passing a StoragePolicy during the initiateMPU phase and then reuses this StoragePolicy for every part uploaded. To ensure upgrade semantics are followed and all OMs, during an upgrade, apply the MPU log consistently, StoragePolicy on an MPU key is ignored when the cluster is not finalized.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16677

How was this patch tested?

  1. Unit Tests and Integration Tests

Generated with Claude Code: Sonnet-5.5

@github-actions github-actions Bot added om client s3 S3 Gateway labels Oct 19, 2026
@anuragp010
anuragp010 marked this pull request as ready for review October 19, 2026 11:17
@anuragp010

Copy link
Copy Markdown
Contributor Author

PTAL @devmadhuu , @priyeshkaratha , @Gargi-jais11 and @xichen01 !

@devmadhuu devmadhuu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for the patch. Can you pls check the review comments ?

@Gargi-jais11 Gargi-jais11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for working on this. Left few comments

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for the contribution. Added a few comments.

@rakeshadr

Copy link
Copy Markdown
Contributor

General Comment:-

@anuragp010. Thanks for adding UT cases.
Please add a few basic MPU e2e integration tests using MiniOzoneCluster. This would help to catch integration issues later during branch reset etc.

@anuragp010
anuragp010 marked this pull request as draft November 6, 2026 10:19
@anuragp010

Copy link
Copy Markdown
Contributor Author

Thanks @rakeshadr for the review !

Please add a few basic MPU e2e integration tests using MiniOzoneCluster. This would help to catch integration issues later during branch reset etc.

The following tests in the PR run on MiniOzoneCluster:

  1. TestOzoneStoragePolicy#testMultipartUploadStoragePolicy - e2e test of MPU with storage policy.
  2. TestOzoneStoragePolicy#testMultipartUploadKeepsStoragePolicyOfInitiate - initiateMPU starts when bucket policy is WARM, but before uploading first part, the bucket policy changes. This test verifies the blocks of each part and the key still land on WARM.
  3. TestOzoneStoragePolicy#testMultipartUploadOverwriteWithDifferentStoragePolicy - Verifies the MPU key's storagePolicy is same as it's blocks in the case of an overwrite.
  4. TestOMMpuStoragePolicyUpgrade - Verifies rejection of initiateMPU requests with storagePolicy when cluster is in pre-finalization and success of requests with storagePolicy post finalization.

@devmadhuu devmadhuu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for updating the patch, LGTM, just few nits. Pls take care.

@anuragp010
anuragp010 requested a review from devmadhuu November 10, 2026 09:43

@devmadhuu devmadhuu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for updating the patch and addressing the review comments. Largely looks good to me, except this below scenario. Can you check if this is possible ?

  1. The cluster is upgraded and finalized. An administrator sets the bucket policy to COLD.
  2. An old client starts an upload without sending any storage-policy field.
  3. OM inherits COLD from the bucket and saves it on the upload.
  4. While that upload is running, the administrator changes the bucket to WARM.
  5. The same old client requests another block, again without a policy field.
  6. OM currently resolves that missing field from the bucket’s current WARM policy, instead of the upload’s saved COLD policy.

@anuragp010

Copy link
Copy Markdown
Contributor Author

except this below scenario. Can you check if this is possible ?

  1. The cluster is upgraded and finalized. An administrator sets the bucket policy to COLD.
  2. An old client starts an upload without sending any storage-policy field.
  3. OM inherits COLD from the bucket and saves it on the upload.
  4. While that upload is running, the administrator changes the bucket to WARM.
  5. The same old client requests another block, again without a policy field.
  6. OM currently resolves that missing field from the bucket’s current WARM policy, instead of the upload’s saved COLD policy.

Hi @devmadhuu , as discussed, yes this is possible. Verified the same. This behaviour is possible in the normal createKey too and will use HDDS-17374 to handle this scenario.

@anuragp010
anuragp010 marked this pull request as ready for review November 13, 2026 05:17

@devmadhuu devmadhuu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for addressing all review comments. LGTM +1. Will review your follow up JIRA PRs as mentioned in this PR review discussion. This PR change is good.

@anuragp010

Copy link
Copy Markdown
Contributor Author

Thanks @devmadhuu !

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anuragp010 added one more comment. Thanks for the continuous effort!

@anuragp010
anuragp010 marked this pull request as draft November 16, 2026 09:21

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM. Thanks @anuragp010 for working on this.

@anuragp010

Copy link
Copy Markdown
Contributor Author

Thanks @rakeshadr !

@priyeshkaratha
priyeshkaratha marked this pull request as ready for review November 17, 2026 04:39

@Gargi-jais11 Gargi-jais11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for updating the PR.
LGTM, +1

@devmadhuu

Copy link
Copy Markdown
Contributor

Thanks @anuragp010 for the patch. Thanks @Gargi-jais11 @rakeshadr for the review.

@devmadhuu
devmadhuu merged commit 3e0f7af into apache:HDDS-11233 Nov 17, 2026
125 of 128 checks passed
@anuragp010

Copy link
Copy Markdown
Contributor Author

Thanks @devmadhuu , @Gargi-jais11 , @rakeshadr for the review !

@anuragp010
anuragp010 deleted the HDDS-16677 branch November 17, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants