Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for benchmarking zero-copy bidirectional reads and memoryview-based downloads in GCS performance microbenchmarks. It adds configuration options, updates parameter structures, integrates environment variable controls, and updates benchmark execution paths to utilize these features. Additionally, it dynamically formats the benchmark results display table. A critical issue was identified in packages/google-cloud-storage/tests/perf/microbenchmarks/time_based/reads/test_reads.py where the os module is used to set environment variables in _worker_init but is not imported, which would result in a NameError at runtime.
Contributor
Author
|
/gcbrun |
yuxin00j
marked this pull request as ready for review
November 5, 2026 08:41
Contributor
Author
|
/gcbrun |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds cases for the opt-in zero-copy
BidiReadObjectResponseparser (GCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READ=1) from #18627 to both read microbenchmark suites (tests/perf/microbenchmarks/reads/andtests/perf/microbenchmarks/time_based/reads/), as requested in its review:write()destination (BytesIOinreads/test_reads.pyand_DummyListBufferintime_based/reads/test_reads.py) and runs in three cases: the baseline with the parser off,_zero_copywith the parser on andaccepts_memoryview=False, and_zero_copy_memoryviewwith the parser on andaccepts_memoryview=True.(zero_copy=False, accepts_memoryview=True)are skipped.Both suites run with default CRC32C verification enabled.
_zero_copy_memoryviewpassesmemoryviewchunks towrite()when the installedgoogle-crc32cacceptsmemoryview(main), andbyteschunks otherwise (for examplegoogle-crc32c1.9.0).Changes
tests/perf/microbenchmarks/_utils.py:zero_copy_envsets and restoresGCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READaround each benchmark, andpublish_benchmark_extra_inforecordszero_copy,accepts_memoryview, andcrc32c_memoryview_supportedinbenchmark.extra_info.reads/andtime_based/reads/:config.yamlandconfig.pyaddzero_copyandaccepts_memoryview(withZERO_COPYandACCEPTS_MEMORYVIEWenvironment overrides intime_based/reads/config.py);test_reads.pypassesaccepts_memoryviewtodownload_rangesand setsGCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READin spawned worker processes.reads/test_reads.py, and from 6 to 12 intime_based/reads/test_reads.py(2 to 6 ingcs-python-benchmark).cloudbuild/display_benchmark_results.py: sizes table columns to fit the rendered rows so the longer case names stay aligned.Testing
I ran both suites locally with
pytest-benchmark(including spawned worker processes,json_to_csv.py, andcloudbuild/display_benchmark_results.pyon the time-based suite) against an in-processgrpc.aioserver implementingBidiReadObject, once withgoogle-crc32c1.9.0 and once with agoogle-crc32cbuild frommainthat acceptsmemoryview. The run also checked case collection, environment variable overrides and restoration, and thatwrite()receivesmemoryviewonly in_zero_copy_memoryviewwhengoogle-crc32csupports it andbytesotherwise.