Skip to content

test(storage): cover zero-copy BidiReadObject reads in the read microbenchmarks - #18993

Open
yuxin00j wants to merge 1 commit into
googleapis:mainfrom
yuxin00j:storage-zero-copy-benchmarks
Open

yuxin00j wants to merge 1 commit into
googleapis:mainfrom
yuxin00j:storage-zero-copy-benchmarks

Conversation

@yuxin00j

@yuxin00j yuxin00j commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds cases for the opt-in zero-copy BidiReadObjectResponse parser (GCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READ=1) from #18627 to both read microbenchmark suites (tests/perf/microbenchmarks/reads/ and tests/perf/microbenchmarks/time_based/reads/), as requested in its review:

  • Every Rapid read workload keeps its existing write() destination (BytesIO in reads/test_reads.py and _DummyListBuffer in time_based/reads/test_reads.py) and runs in three cases: the baseline with the parser off, _zero_copy with the parser on and accepts_memoryview=False, and _zero_copy_memoryview with the parser on and accepts_memoryview=True.
  • Regional workloads and (zero_copy=False, accepts_memoryview=True) are skipped.

Both suites run with default CRC32C verification enabled. _zero_copy_memoryview passes memoryview chunks to write() when the installed google-crc32c accepts memoryview (main), and bytes chunks otherwise (for example google-crc32c 1.9.0).

Changes

  • tests/perf/microbenchmarks/_utils.py: zero_copy_env sets and restores GCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READ around each benchmark, and publish_benchmark_extra_info records zero_copy, accepts_memoryview, and crc32c_memoryview_supported in benchmark.extra_info.
  • reads/ and time_based/reads/: config.yaml and config.py add zero_copy and accepts_memoryview (with ZERO_COPY and ACCEPTS_MEMORYVIEW environment overrides in time_based/reads/config.py); test_reads.py passes accepts_memoryview to download_ranges and sets GCS_PYTHON_CLIENT_ZERO_COPY_BIDI_READ in spawned worker processes.
  • Collected cases go from 68 to 136 in reads/test_reads.py, and from 6 to 12 in time_based/reads/test_reads.py (2 to 6 in gcs-python-benchmark).
  • cloudbuild/display_benchmark_results.py: sizes table columns to fit the rendered rows so the longer case names stay aligned.

Testing

cd packages/google-cloud-storage
ruff check --select I cloudbuild/display_benchmark_results.py tests/perf/microbenchmarks
ruff format --check --target-version=py310 cloudbuild/display_benchmark_results.py tests/perf/microbenchmarks
flake8 cloudbuild/display_benchmark_results.py tests/perf/microbenchmarks

I ran both suites locally with pytest-benchmark (including spawned worker processes, json_to_csv.py, and cloudbuild/display_benchmark_results.py on the time-based suite) against an in-process grpc.aio server implementing BidiReadObject, once with google-crc32c 1.9.0 and once with a google-crc32c build from main that accepts memoryview. The run also checked case collection, environment variable overrides and restoration, and that write() receives memoryview only in _zero_copy_memoryview when google-crc32c supports it and bytes otherwise.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for benchmarking zero-copy bidirectional reads and memoryview-based downloads in GCS performance microbenchmarks. It adds configuration options, updates parameter structures, integrates environment variable controls, and updates benchmark execution paths to utilize these features. Additionally, it dynamically formats the benchmark results display table. A critical issue was identified in packages/google-cloud-storage/tests/perf/microbenchmarks/time_based/reads/test_reads.py where the os module is used to set environment variables in _worker_init but is not imported, which would result in a NameError at runtime.

@yuxin00j

yuxin00j commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@yuxin00j
yuxin00j marked this pull request as ready for review November 5, 2026 08:41
@yuxin00j
yuxin00j requested review from a team as code owners November 5, 2026 08:41
@yuxin00j

yuxin00j commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@yuxin00j
yuxin00j requested a review from chandra-siri November 5, 2026 08:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant