Skip to content

fix(storage): offload CRC32C verification by default only for zero-copy reads - #19095

Open
yuxin00j wants to merge 1 commit into
googleapis:mainfrom
yuxin00j:storage-offloop-opt-in
Open

yuxin00j wants to merge 1 commit into
googleapis:mainfrom
yuxin00j:storage-offloop-opt-in

Conversation

@yuxin00j

Copy link
Copy Markdown
Contributor

Summary

This narrows the off-loop CRC32C verification from #18628 before it ships, so it is on by default only for zero-copy reads.

  • Default: read chunks of 1 MiB or more that reach a write() destination as memoryview (the zero-copy parser with a google-crc32c that accepts memoryview, and accepts_memoryview=True) are verified on the worker pool, as in perf(storage): verify CRC32C of large read chunks off the event-loop thread #18628. Every other chunk is verified inline again.
  • Override: GCS_PYTHON_CLIENT_CRC32C_OFFLOAD=1 offloads every chunk of 1 MiB or more outside whole-object ranges, and GCS_PYTHON_CLIENT_CRC32C_OFFLOAD=0 none.

Offloading pays off once no bytes copy of each chunk is left on the event-loop thread. With such a copy (the generated parser or accepts_memoryview=False), results are mixed: SDK multi-stream reads and gcsfs cat_ranges still gain, but gcsfs read of 1 MiB or more loses in 10 of 18 cases with the generated parser.

Changes

  • GCS_PYTHON_CLIENT_CRC32C_OFFLOAD replaces GCS_PYTHON_CLIENT_CRC32C_OFFLOAD_MIN_BYTES, which never shipped.
    • Unset or empty selects the default. 1, true, yes or on offloads every chunk of 1 MiB or more outside whole-object ranges, and any other value none.
    • It is read when a chunk arrives, so a library can set it before it opens a stream.
  • Each download records whether its destination takes chunks unwrapped (a write() destination with accepts_memoryview=True). Buffer destinations still copy each chunk on the loop thread, so they stay inline by default.
  • The threshold is fixed at google-crc32c's GIL-release threshold (1 MiB). Smaller chunks would hold the GIL on the worker.
  • The enable_checksum docstring of download_ranges describes the default and the override.
  • test_mrd_zero_copy_bidi_read sets the flag to 1, so the Rapid system test covers the offload path even with a google-crc32c that rejects memoryview.

Testing

cd packages/google-cloud-storage
ruff check --select I --target-version=py310 --line-length=88 docs google tests noxfile.py setup.py
ruff format --check --target-version=py310 --line-length=88 docs google tests noxfile.py setup.py
flake8 google tests
pytest tests/unit tests/resumable_media/unit
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python pytest tests/unit/asyncio

@yuxin00j
yuxin00j requested review from a team as code owners November 16, 2026 10:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refines the CRC32C checksum offloading logic in the asyncio multi-range downloader. Instead of offloading all large chunks, it now defaults to offloading only when chunks reach their destination as a memoryview, with an override available via the GCS_PYTHON_CLIENT_CRC32C_OFFLOAD environment variable. The review feedback suggests optimizing the unit tests by caching patchers in _create_view_response to prevent nested patching and redundant cleanup registrations when the helper is called multiple times in a single test.

…py reads

Read chunks of 1 MiB or more are hashed on the CRC32C worker pool by
default only when they reach a write() destination as a memoryview: the
zero-copy parser with a google-crc32c that accepts memoryview, and
accepts_memoryview=True. Every other chunk is verified inline on the
event-loop thread again, so callers that have not opted in to zero-copy
reads keep verifying inline. Buffer destinations copy each chunk on the
loop thread, so they stay inline by default too.

GCS_PYTHON_CLIENT_CRC32C_OFFLOAD overrides the default: 1 (or true, yes,
on) offloads every chunk of 1 MiB or more outside whole-object ranges,
and any other non-empty value none. The variable is read when a chunk
arrives, so a library can set it before it opens a stream. The threshold
is fixed at google-crc32c's GIL-release threshold (1 MiB), and
GCS_PYTHON_CLIENT_CRC32C_OFFLOAD_MIN_BYTES is removed.

In 5-round SDK and gcsfs read microbenchmarks, offloading memoryview
chunks gained a median 35% on SDK reads with 4-16 streams per event loop
and 30% on gcsfs cat_ranges. When each chunk was still copied on the
loop thread, results were mixed: SDK reads with 4-16 streams per loop
still gained 10% with the generated parser, but gcsfs reads of 1 MiB or
more lost in 10 of 18 cases, and single-threaded gcsfs reads lost a
median 7% with accepts_memoryview=False.
@yuxin00j
yuxin00j force-pushed the storage-offloop-opt-in branch from 6f87228 to 9147a64 Compare November 16, 2026 12:39
@yuxin00j

Copy link
Copy Markdown
Contributor Author

/gcbrun

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant