[jayzuccarelli]

First of three PRs for #399, one benchmark each as suggested there. This one adds the shared piece (an io group that is not part of default) together with the first benchmark that uses it.

What it measures

io_whole_file reads and writes a whole file in a single call, which is the pattern gh-120754 optimized:

Each result measures a single API:

• read_bytes_{4k,1m}: open(path, "rb").read()
• write_bytes_{4k,1m}: open(path, "wb").write(data)
• read_text_1m / write_text_1m: Path.read_text() / Path.write_text(), utf-8, ASCII content
• read_bytes_{4k,1m}_uncached: open(path, "rb").read() with the file's pages dropped from the page cache first

Path.read_bytes() / write_bytes() are thin wrappers over the same open() calls, so they are not measured separately.

8 results on Linux (6 where os.posix_fadvise is missing). 4 KiB is where per-call overhead and the number of system calls dominate; 1 MiB is where buffer allocation and copying dominate. I had an 8 MiB size too, but it scaled linearly from 1 MiB and sits closer to dirty page throttling, so I dropped it to keep the result count small.

Apart from the uncached reads below, everything runs in a fresh temp dir with a warm page cache and no fsync, so it measures Python's I/O stack and syscall overhead rather than the disk. Writes always create a new file (unlinked outside the timed region): truncating and rewriting an existing file on ext4 triggers writeback on close (auto_da_alloc), which made a 16 MiB rewrite take ~490 ms +- 220 ms instead of ~30 ms. The run records io.DEFAULT_BUFFER_SIZE in the metadata.

Uncached reads: read_bytes_{4k,1m}_uncached time the same open(path, "rb").read() with the file's pages dropped from the page cache before each read (os.posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED), outside the timed region; the file is fsync'ed once in setup since DONTNEED only drops clean pages, no root needed). fincore shows 0 resident pages after every drop. On a SATA SSD (ext4, 3.14) 4 KiB goes from ~17 us cached to ~204 us uncached and 1 MiB from ~148 us to ~3.4 ms. They mostly track the storage device, so they are noisier and named apart from the cached results. They are skipped where os.posix_fadvise is missing (macOS, Windows; macOS F_NOCACHE doesn't evict already-cached pages). Writes stay cached only, since an uncached write needs an fsync in the timed region, which measures the device's flush path rather than Python. Large files are left for a follow-up.

Setup allocates and frees one 8 MiB block before anything is measured. On glibc that raises malloc's dynamic mmap/trim thresholds above the 1 MiB working set; without it read_text_1m (which frees two 1 MiB blocks per call) sat right at the trim threshold and, depending on heap history, flipped between ~430 us and ~1.3 ms on the same interpreter because the heap top was handed back to the kernel and faulted in again every call.

Why opt-in

Per the issue, I/O benchmarks shouldn't run by default. The benchmark is tagged io, [group io] picks it up through the tag, and [group default] excludes it by name (a -io group exclusion is silently ignored by the manifest parser, so it has to name the benchmark). pyperformance list is unchanged at 100 benchmarks; -b io selects it.

Sample timings (pyperformance run -b io_whole_file --fast, one untuned Linux box, ext4 /tmp)

result	3.13.13	3.14.4
io_whole_file_read_bytes_4k	15.6 us	13.8 us
io_whole_file_write_bytes_4k	34.0 us	34.2 us
io_whole_file_read_bytes_1m	153 us	148 us
io_whole_file_write_bytes_1m	912 us	916 us
io_whole_file_read_text_1m	437 us	501 us
io_whole_file_write_text_1m	1.15 ms	1.12 ms

How to run

python -m pyperformance run -b io
python -m pyperformance run -b io_whole_file

Tests

• python -m pyperformance.tests passes (it runs -b all, so the io benchmarks execute there too)
• pyperformance list unchanged at 100, list -b io is just io_whole_file
• flake8 with the tox config is clean on the new file
• uncached skip path checked with os.posix_fadvise removed (6 cached results, no crash)
• No version bump: 1.14.0 is already tagged, so the changelog line goes under a new 1.15.0 heading. Happy to bump __version__ too if you want that in the same PR, as Add bm_base64 covering common base64 module APIs #447 did.

Part of #399.

AI disclosure: I used Claude Code to write the benchmark and docs; I reviewed the code and ran it.

[cmaloney]

Can you explore ways to make this measure

1. Cached vs. uncached I/O. Reading or writing in a loop doesn't actually touch disk as the OS realizes it's unnecessary. One of the things that is important to measure is what happens if we actually get a not cached read (which is why I/O operations historically dropped the GIL while executing)
2. Sizes up to very largefiles. This is tested in CPYthon but it woudl be good to go up to 4+GB files here.

For the largefiles that is probably a second opt in (cpython's core tests do that), although needs some design work

[jayzuccarelli]

Sure. Rough plan:

1. Uncached: after writing and fsyncing the file, drop its pages each iteration with os.posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED). It needs no root; here a 256 MiB read goes from 145 ms cached to 626 ms uncached, so it is really reaching the disk. macOS could use F_NOCACHE; I don't see an unprivileged equivalent on Windows that keeps normal buffered I/O, so it would skip there. Since these numbers mostly track the device, I'd give them separate names (e.g. io_whole_file_uncached) so they don't mix with the cached ones.

2. Large files: a second opt-in group, like CPython's largefile resource, with sizes just past 2 GiB and 4 GiB since that is where size handling tends to break. It needs ~5 GB of free disk and enough memory for readall, and would skip with a message otherwise.

Would you rather keep this PR as the cached baseline and do uncached and largefiles as follow-up PRs, one per benchmark like before?

[cmaloney]

I think building out the core primitives and how understandable the results are are important in the first PR. Need to do it with straightforward cases to make sure it's working right before we do more complicated ones.

[jayzuccarelli]

Added uncached reads as io_whole_file_read_bytes_4k_uncached and io_whole_file_read_bytes_1m_uncached: before each timed open().read() the file's pages are dropped with posix_fadvise(POSIX_FADV_DONTNEED) outside the timer, and fincore confirms 0 resident pages after every drop. Here (SATA SSD, ext4) 4 KiB goes from 17 us to 204 us and 1 MiB from 148 us to 3.4 ms; the uncached ones are noisier because they track the device, and they skip where posix_fadvise is missing (macOS, Windows). I left writes cached only, since an uncached write means an fsync inside the timed region, which measures the device's flush path rather than Python; large files stay for a follow-up.

[jayzuccarelli]

@cmaloney when you have a moment: uncached reads are in as io_whole_file_read_bytes_4k_uncached and _1m_uncached, per your cached vs uncached note. Happy to adjust the approach.

[cmaloney]

I don't think this will do enough, I suspect we need https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html#drop-caches for linux and other platform-specific answers for Windows and macOS (OSes beyond that would be nice, but that would cover the PEP 11 tier 1).

When changing this want to see local numbers which demonstrate that the behavior clearly changes.

[jayzuccarelli]

Measured on Linux 5.15, ext4, SATA SSD, untuned, 3 default runs:

	cached	uncached
4k	15.5 us ± 0.5	201 us ± 26
1m	143 us ± 10	3.33 ms ± 0.34

Residency (mincore after every drop): steady state is 0 pages resident for both sizes. The one exception is the first drop of a small file right after fsync, which the kernel ignores; under pyperf that happened once per worker, always the first drop, inside the discarded warmup (21 of 87,039 drops). Without the fsync, DONTNEED drops nothing.

drop_caches needs root and flushes every clean page on the machine, once per drop across ~20 workers per result. It also only frees clean pages, so it needs the same fsync.

macOS: F_NOCACHE disables caching for one fd; Apple doesn't document whether it evicts already-cached pages. Windows: FILE_FLAG_NO_BUFFERING needs CreateFile plus sector-aligned buffers, so it can't time open().read(). My plan is to skip uncached on both with a visible message. OK?

[methane]

If we want to measure disk performance, we should use a dedicated tool. pyperformance is a tool designed to measure Python performance. I don't think we should add an uncached benchmark that is heavily affected by disk speed.

[cmaloney]

Lets keep discussion on the PR about the PR contents, not about meta design. Happy to discuss tradeoffs of building a new suite of benchmarks vs adding opt-in ones in the issue thread #399. These are not enabled by default and will not effect people running pyperformance unless they choose specific flags.