Skip to content

Handle short destination writes in stream writers - #371

Open
akshitkhokhani wants to merge 1 commit into
indygreg:mainfrom
akshitkhokhani:codex/zstandard-short-write-validation
Open

akshitkhokhani wants to merge 1 commit into
indygreg:mainfrom
akshitkhokhani:codex/zstandard-short-write-validation

Conversation

@akshitkhokhani

Copy link
Copy Markdown

Stream writers currently discard the byte count returned by their destination's write(). A destination accepting a short write silently loses the rest of the output, even though the outer writer reports consuming the input.

Retry the unwritten suffix in the C, CFFI and Rust compression/decompression writers, including compression flush. Preserve legacy callback destinations that return None after consuming the whole chunk. Zero progress raises BlockingIOError; invalid counts are rejected, and destination exceptions propagate. The documentation states this contract and excludes general nonblocking or post-failure resumability guarantees. copy_stream() is unchanged.

Fixes #370.

Validation on macOS ARM64 / CPython 3.13.7, using source-built backends and synthetic local data with network access disabled:

  • Before the fix: the two short-write regressions fail on both C and CFFI; two legacy None controls pass on each.
  • After the fix: both complete stream-writer test modules pass on C, CFFI and Rust — 45 tests plus 12 subtests per backend. Coverage includes compression write/flush, decompression, zero/negative/oversized/noninteger counts, the index protocol, output accounting and exception identity.
  • Independent agent review reran all three backends and additional mixed short-count/None and index-exception controls; no blocking findings remained.
  • Repository-wide Ruff and git diff --check pass. Rust changes were formatted with rustfmt. The full project/slow test suites, other Python versions and other operating systems were not run locally.

AI assistance was used for implementation, tests and independent review; no maintainer approval is implied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream writers silently lose output when a destination accepts a short write

1 participant