[kpumuk]

Streaming request bodies with checksum trailers creates temporary Strings for input chunks, interpolated framing, and slices of the encoded buffer. For a 100 MiB upload, these accumulate roughly 301 MiB of String memory when GC is disabled.

This change reduces allocations in AwsChunkedTrailerDigestIO through three optimizations:

1. Initialize @chunk with ''.b and reuse it through @io.read(@base_chunk_size, @chunk).
2. Append chunk framing and trailer components directly to @encoded_buffer, avoiding intermediate interpolated Strings.
3. Append the encoded buffer directly when consuming it completely, avoiding the temporary String created by slice!. Partial reads use byteslice and retain the remaining suffix; they can still copy data.

Benchmark

Measured with a 100 MiB binary file, CRC32, 16 KiB encoding chunks, and Ruby 4.0.6 on arm64 macOS. Each row adds one optimization to the previous row. These measurements were captured during development, before squashing the commits.

Case	Memory	Allocations	Peak RSS	Streaming time	GC cycles
Original code	301.11 MiB	25,652	97.94 MiB	116.99 ms	7
Reuse buffer	200.87 MiB	19,247	94.80 MiB	106.95 ms	6
+ Avoid interpolation	100.57 MiB	12,842	77.08 MiB	100.42 ms	3
+ Avoid slice!	0.27 MiB	6,438	22.08 MiB	88.96 ms	0

• Memory: growth in memory occupied by Strings with GC disabled, measured using ObjectSpace; not total process memory or cumulative allocator traffic.
• Allocations: number of Strings allocated during that GC-disabled run.
• Peak RSS: maximum resident process memory, including startup and fixture preparation.
• Streaming time: encoding and hashing the output into a discard sink; excludes network transfer.
• GC cycles: collections during streaming.

Peak RSS, streaming time, and GC cycles are medians of five fresh processes with normal GC. Memory and allocations come from a separate GC-disabled run. Results vary with Ruby version, platform, system load, and benchmark overhead.

Reproduce

Save the script below as /tmp/checksum_benchmark.rb and run it from the repository root:

ruby /tmp/checksum_benchmark.rb

The script benchmarks only the checked-out implementation. Run it on the base and feature branches to compare them. It uses Ruby’s standard library, local SDK code, and /usr/bin/time on macOS or Linux.

It prints all five metrics and verifies identical encoded lengths and SHA-256 hashes across its six runs.

Standalone Ruby benchmark

# frozen_string_literal: true
# Run from the repository root: ruby /tmp/checksum_benchmark.rb
require 'base64'
require 'digest'
require 'json'
require 'objspace'
require 'open3'
require 'rbconfig'
require 'tempfile'
require 'zlib'

if ARGV.empty?
  mac = RUBY_PLATFORM.include?('darwin')
  abort 'Requires macOS or Linux with /usr/bin/time' unless mac || RUBY_PLATFORM.include?('linux')
  results = (%w[runtime] * 5 + ['allocations']).map do |mode|
    out, err, status = Open3.capture3(
      { 'LC_ALL' => 'C' }, '/usr/bin/time', mac ? '-l' : '-v',
      RbConfig.ruby, File.expand_path(__FILE__), mode
    )
    abort err unless status.success?
    result = JSON.parse(out)
    rss = err.match(mac ? /^\s*(\d+)\s+maximum resident set size$/ : /Maximum resident set size \(kbytes\):\s*(\d+)/)
    abort 'Peak RSS unavailable' unless rss
    result['rss'] = rss[1].to_i * (mac ? 1 : 1024)
    result
  end
  abort 'Output mismatch' unless results.map { |r| r.values_at('bytes', 'sha256') }.uniq.size == 1
  profile = results.pop
  abort 'GC ran during allocation profiling' unless profile['gc'].zero?
  median = ->(key) { results.map { |r| r.fetch(key) }.sort[2] }
  puts RUBY_DESCRIPTION
  puts '| Memory | Allocations | Peak RSS | Streaming time | GC cycles |'
  puts '|---:|---:|---:|---:|---:|'
  puts format('| %.2f MiB | %d | %.2f MiB | %.2f ms | %d |',
              profile['memory'] / 1048576.0, profile['strings'],
              median.call('rss') / 1048576.0, median.call('seconds') * 1000, median.call('gc'))
  exit
end

$LOAD_PATH.unshift(File.expand_path('gems/aws-sdk-core/lib'))
%w[handler_builder plugin handler].each { |name| require "seahorse/client/#{name}" }
require 'aws-sdk-core/plugins/checksum_algorithm'

class Sink
  attr_reader :bytes, :digest
  def initialize
    @bytes = 0
    @digest = Digest::SHA256.new
  end
  def write(chunk)
    @digest.update(chunk)
    @bytes += chunk.bytesize
    chunk.bytesize
  end
end

Tempfile.create('checksum-benchmark') do |file|
  file.binmode
  block = (0..255).to_a.pack('C*') * 4096
  100.times { file.write(block) }
  file.flush
  block = nil
  body = Aws::Plugins::ChecksumAlgorithm::AwsChunkedTrailerDigestIO.new(
    io: file, algorithm: 'CRC32', location_name: 'x-amz-checksum-crc32'
  )
  sink = Sink.new
  body.read(32_768)
  body.rewind
  GC.start
  GC.disable if ARGV.first == 'allocations'
  strings = ObjectSpace.count_objects[:T_STRING]
  memory = ObjectSpace.memsize_of_all(String)
  gc = GC.stat
  started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  copied = IO.copy_stream(body, sink)
  seconds = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
  cycles = GC.stat[:count] - gc[:count]
  memory = ObjectSpace.memsize_of_all(String) - memory
  strings = ObjectSpace.count_objects[:T_STRING] - strings
  GC.enable
  raise 'Encoded size mismatch' unless copied == body.size && copied == sink.bytes
  puts JSON.generate(memory: memory, strings: strings, gc: cycles, seconds: seconds,
                     bytes: sink.bytes, sha256: sink.digest.hexdigest)
end

Generated with AI assistance and reviewed by @kpumuk

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[czj]

I ran this branch against the released AwsChunkedTrailerDigestIO (aws-sdk-core 3.260.0) on Ruby 4.0.9, arm64 macOS.

A differential fuzz of 9,000 cases gives byte-identical output, and #size still matches the bytes read. The cases mix random binary payloads up to 300 KB, chunk sizes from 16 B to 16 KiB, CRC32, SHA1 and SHA256, varied read lengths, reads with and without a reused output buffer, and rewinds mid-stream.

Streaming 100 MiB through it with 16 KiB reads into a reused buffer, as Net::HTTP does, median of 5:

	3.260.0	this branch
time	49 ms	20 ms
allocated objects	26,240	6,472
GC runs	9	0
RSS growth	+99.5 MB	+0.0 MB

It also complements #3465, which removes the same kind of per-chunk copies when the checksum goes in a header.

[jterapin]

Thanks @kpumuk for the contribution and @czj for the additional testing and complementary work in #3465! The allocation reductions look promising. 🤩

I plan to review both PRs together since they both cover similar path. I'll need more time to assess any potential breaking changes.

Stay tuned! 👍

[czj]

Two small follow-ups, in case they help the review. Happy to drop either.

Breaking-change surface. As far as I can tell, the only observable change is that the wrapped body now receives read(length, buffer) instead of read(length). Bodies already need to accept that call: without a trailer, Net::HTTP streams request.body through IO.copy_stream, which calls read(len, buf) (or readpartial(len, buf)) on any non-IO source. The SDK's own IO-likes accept a buffer too (Query::ParamList::IoValue#read, S3::FilePart#read). The one edge case: since checksums became the default, a custom body that only defines read(length) currently works on the trailer path thanks to this wrapper. It would now raise ArgumentError. That seems unlikely, but it's worth stating.

The last allocation per chunk. The ~6,400 Strings left for 100 MiB are one per 16 KiB chunk: @chunk.bytesize.to_s(HEX_BASE). Every chunk except the last has the same size, so its header can be built once:

           @base_chunk_size = @chunk_size - @overhead_bytes
+          @full_chunk_header = "#{@base_chunk_size.to_s(HEX_BASE)}\r\n".freeze
           @chunk = ''.b

-              @encoded_buffer << @chunk.bytesize.to_s(HEX_BASE) << "\r\n" << @chunk << "\r\n"
+              @encoded_buffer << chunk_header << @chunk << "\r\n"

        def chunk_header
          return @full_chunk_header if @chunk.bytesize == @base_chunk_size

          "#{@chunk.bytesize.to_s(HEX_BASE)}\r\n"
        end

On a standalone copy of the class, streaming 100 MiB with 16 KiB reads into a reused buffer (Ruby 4.0.9, arm64 macOS) gives:

	this branch	+ cached header
allocated objects	6,411	9
time (median of 5)	55.7 ms	54.3 ms

The gain is allocations only, not time. A 2,000-case fuzz across random payloads, chunk sizes and read lengths produced byte-identical output.

#3465 and this PR. Both touch checksum_algorithm.rb, its spec and the CHANGELOG, but not the same methods. Whichever lands first, I'll rebase #3465 on top.

[jterapin]

Thanks @kpumuk for the allocation improvements and @czj for the additional testing and suggestions!

I've added some backwards compatible changes for V3. For V4, I think we can require the IO interface to support read(length, buffer).

@kpumuk, could you rerun your benchmarks against the updated branch to confirm that the compatibility changes retain the allocation and performance improvements?

@czj - caching the chunk header looks like a worthwhile optimization! Could you open a separate PR for it? I’ll make a few minor changes to your current PR (#3465), then ask the team to review both PRs in conjunction.

[kpumuk]

Re-ran the 100 MiB benchmark on e403f9052c. The buffered path retains the same memory and allocation improvements, with no meaningful timing change observed.

Ruby 4.0.6, CRC32, 16 KiB chunks:

Case	Memory	Allocations	Peak RSS	Streaming time	GC cycles
Original code	301.11 MiB	25,653	96.77 MiB	109.06 ms	7
Previous PR revision (3db5d32ba0)	0.27 MiB	6,439	21.92 MiB	82.51 ms	0
Current, buffer enabled	0.27 MiB	6,439	21.83 MiB	82.98 ms	0
Current, buffer disabled	100.50 MiB	12,844	62.64 MiB	105.67 ms	3

Memory and String allocations were measured with GC disabled. RSS, streaming time, and GC cycles are medians of 15 fresh processes per case, with execution order rotated.

[czj]

@jterapin the chunk header is in #3490, on top of #3465 so the three can be reviewed together: 6,441 → 39 allocations on the same 100 MiB upload, no change in time.

[jterapin]

The manual Build SDK dry run passed via validation PR #3488: successful run.

🤖 This comment was drafted with AI assistance for @jterapin. ✨

[ashishdhingra]

Verified fixes and specs locally.