[yuxin00j]

Summary

This makes the off-loop CRC32C verification from googleapis#18628 opt-in before it ships.

• Default: every chunk is verified inline again, and the worker pool never starts.
• Opt in: with GCS_PYTHON_CLIENT_CRC32C_OFFLOAD=1, read chunks of 1 MiB or more are verified on the worker pool, as in perf(storage): verify CRC32C of large read chunks off the event-loop thread googleapis/google-cloud-python#18628.

Offloading helps most when several streams share one event loop and chunks reach the destination without a bytes copy (the zero-copy parser with a google-crc32c that accepts memoryview, and accepts_memoryview=True or a buffer destination). With one stream per loop it shows no consistent gain, and with the generated parser or accepts_memoryview=False, gcsfs loses in 9 of 18 read cases:

Rapid reads, c4-standard-192, median MiB/s	Offload off	Offload on
gcsfs read, seq 16p_1t 16 MiB, generated parser	21,520.0	19,617.1 (-8.8%)
gcsfs read, seq 1p_1t 1 MiB, zero-copy, accepts_memoryview=False	1,406.0	1,077.1 (-23.4%)
SDK time_based/reads, seq 8p_16c, 1 stream per process, generated parser	8,619.1	7,551.3 (-12.4%)
SDK time_based/reads, seq 8p_16c, 4 streams per process, zero-copy, accepts_memoryview=True	16,713.9	22,192.2 (+32.8%)
gcsfs read, seq 1p_32t 100 MiB, zero-copy, accepts_memoryview=True	2,046.7	2,786.7 (+36.2%)

The SDK benchmark opens one stream per process. The 4-stream row passes MultiStreamConfig(min_streams=4, max_streams=4).

Changes

• GCS_PYTHON_CLIENT_CRC32C_OFFLOAD replaces GCS_PYTHON_CLIENT_CRC32C_OFFLOAD_MIN_BYTES, which never shipped.
  • It accepts 1, true, yes or on, the same values as GCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READ.
  • It is read at runtime rather than at import, so a library can opt in before it opens a stream.
• The threshold is fixed at google-crc32c's GIL-release threshold (1 MiB). Smaller chunks would hold the GIL on the worker.
• The enable_checksum docstring of download_ranges now describes the opt-in.
• test_mrd_zero_copy_bidi_read opts in, so the Rapid system test still covers the offload path.

Testing

cd packages/google-cloud-storage
ruff check --select I --target-version=py310 --line-length=88 docs google tests noxfile.py setup.py
ruff format --check --target-version=py310 --line-length=88 docs google tests noxfile.py setup.py
flake8 google tests
pytest tests/unit tests/resumable_media/unit
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python pytest tests/unit/asyncio
RUN_RAPID_SYSTEM_TESTS=True RAPID_BUCKET=<rapid-bucket> CROSS_REGION_BUCKET=<rapid-bucket> pytest tests/system/test_rapid.py::test_mrd_zero_copy_bidi_read