[czj]

When an endpoint makes aws-sdk-s3 skip trailer checksums (custom endpoints such as Wasabi or Scaleway, plain HTTP, JRuby), the request checksum goes in a header, so update_in_chunks reads the whole body first. Each io.read(CHECKSUM_CHUNK_SIZE) returns a new 1 MiB String, so a 72 MB upload allocates 72 MiB of temporary Strings before a single byte is sent.

This reads every chunk into the same buffer with io.read(CHECKSUM_CHUNK_SIZE, chunk). Passing a buffer is safe for any body the SDK already accepts: Net::HTTP sends bodies through IO.copy_stream, which calls read(length, buffer) on non-IO objects, and ParamList and the trailer ChecksumIO already take one.

put_object against a stubbed endpoint, 5 PUTs per size:

body	time per PUT	GC runs	RSS
21 MB	9.4–11.4 → 6.9–7.2 ms	2 → 0	+62–66 MB → +3 MB
72 MB	31.6–34.9 → 23.9–28.6 ms	10 → 0	+15–42 MB → +4 MB

The new spec checks the CRC32 of a multi-chunk body and that every read got the same buffer. It fails without the change. checksum_algorithm_spec passes (52 examples). I didn't run the HTTP/2 specs, my local bundle doesn't have http-2. #3448 is about the trailer path, not this one.

Generated by AI tools (Claude Code). Prompted, read, and verified by @czj, a caring human :)

[jterapin]

Thanks @czj! I’ve rebased this branch onto #3448 and added a separate compatibility commit to preserve custom bodies supporting only read(length). The checksum calculation now hashes the bytes returned by the reader rather than assuming the supplied buffer was filled. As I mentioned in other pr, I hope to remove this check in the next MV.

Could you rerun your benchmarks against the updated branch to confirm that the allocation and performance improvements are retained?

[czj]

Thanks @jterapin for the rebase and the compatibility commit! I reran the benchmark on the updated branch, against version-3 at 942d8af, with put_object on a stubbed plain HTTP endpoint so the checksum goes in a header. The gains are all still there:

body	Ruby	time per PUT	GC runs per 5 PUTs	RSS after 5 PUTs
21 MB	interpreter	11.7–12.9 → 9.6–10.2 ms	2 → 0	+67–68 MB → +2–4 MB
21 MB	YJIT	14.0–14.1 → 11.0–11.7 ms	2 → 0	+67–68 MB → +2–4 MB
72 MB	interpreter	38.0–40.4 → 32.3–32.5 ms	8 → 0	+38–39 MB → +1–3 MB
72 MB	YJIT	41.8–42.3 → 33.8–35.1 ms	8 → 0	+40–41 MB → +2–3 MB

Times are the median of 5 PUTs, ranges over 3 runs.

Thanks @ashishdhingra for the review, all three are in 2f3c5a3: fill_encoded_buffer reads into a local so @chunk stays the reusable buffer, with a spec where the reader returns a frozen String (it raises FrozenError without the change), buffer_enabled is consumed with options.delete(...) || false, and the trailer examples are skipped on JRuby. aws-sdk-core specs pass locally (2,515 examples).

These lines come from b708773, which is also on #3448, so #3448 would still have the FrozenError if it lands first. @jterapin would you rather have the fix on #3448 instead? I can push it on a branch on top of chunked-digest for you to pull, and drop it from here.

[jterapin]

@czj
Hi! Yes, I’ll make the changes in #3448 first. You can then rebase this PR, or I can do it, and we’ll go from there. I'll get to this today! :)

[jterapin]

@ashishdhingra Thanks for the review! The shared fixes are now applied in #3448.

@czj I’ve rebased this branch onto the updated #3448.

🤖 This comment was drafted with AI assistance for @jterapin. ✨

[jterapin]

The manual Build SDK dry run passed via validation PR #3489: successful run. This validates the branch rebased onto the updated #3448.

🤖 This comment was drafted with AI assistance for @jterapin. ✨

[ashishdhingra]

Verified fixes and specs locally.