Repository navigation
Conversation
…cannot open (zarr-developers#4604) * feat(zarr-metadata): add a reader that reports what a reader cannot open `read_openable_node_metadata_v3` reads a v3 `zarr.json` as `read_node_metadata_v3` does and adds a problem of the new kind `unclaimed_declaration` for each declaration a reader must understand that nothing in scope claims: an unknown codec, data type or chunk grid, or an unknown top-level member without `"must_understand": false`. Its reading has no problems exactly when a reader that understands the scope can open the node. The other readers keep accepting unclaimed declarations, so the format stays open; their docstrings and the parsing guide now point to `unclaimed_declarations` and the new reader. Prompted by refkv's report that an unknown v3 member parses without error. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): number the changelog fragment for #440 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): number the unopenable-declarations fragment after upstream PR 4604 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… each spec claim (zarr-developers#4605) * docs(zarr-metadata): split the landing page into guide pages and link each spec claim Shrink index.md to a short pitch, one example and a reader-by-reader list of where to go. Move the rest into guide pages: models and readers, what the readers check, consolidated metadata, repairing known writer bugs, writing an extension, JSON Schema and Pydantic. Each claim backed by a Zarr spec links to the spec section that makes it; the consolidated metadata page says what the specs do and do not settle and links the zarr-specs PRs and issues and the original zarr-python implementation. Trim the README so it no longer mirrors the landing page, and say "context" where the pages said "scope". Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): number the guide-pages fragment after its fork PR Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): say "member" for a JSON name/value pair in the guide pages The rename branch keeps "member" for a name/value pair of a metadata object, so the new pages follow: "a top-level member that the spec does not define is an extra field". "Extra field" and "extension field" stay, after the names extra_fields and ZarrV3ExtensionField. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): say "member" for a JSON name/value pair on every page Sweep the parsing-validation and context pages, which said "field" for a name/value pair of a metadata object, and state the one exception on the models page: "extra field" follows the spec's "additional fields" and the names extra_fields, ExtraFieldDefinition and must_understand_fields; "field" otherwise means a field of a Python class, or a part of a structured data type. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): say "member" in the two must_understand_fields sentences The models page said "each unclaimed field" and "any such field" where the landing page and every neighboring sentence say "member". Roborev flagged the mismatch. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): say why the additional-fields link points at the latest spec v3.1.0 has no "Additional fields" heading; PR 373 added it after the release, so the link cannot be pinned. Say so beside the link. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): split the additional-fields sentence and date its claim Roborev found the three-clause sentence hard to parse and its "no tagged release yet" unanchored in time. Two sentences now, anchored to v3.1.0, with a note to pin the link once a release carries the heading. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): fix six wording and example issues roborev found on the guide pages The pydantic example now builds the manifest with model_validate, so it type-checks as the next sentence claims. The consolidated page says three other readers take a context, not two. The equality section quotes the v2 spec on a missing .zattrs in two sentences. The two numbered Sphinx anchors are replaced by line ranges in the spec source at the v3.1.0 tag. The landing page says once that the library does no chunk or store I/O. The extensions page example and the zarr_metadata.definition docstring example are pinned to each other by a test. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(zarr-metadata): make the example-drift test say what it could not find Roborev asked for diagnostics: the test now asserts there is exactly one extensions-session block and exactly one marker line in the docstring, each with a message, and compares line lists so a mismatch points at the differing line. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): build the extension example with union and say "context" throughout On main a Context has no update method, so the acme.lz4 example on the extensions page now adds its definition with union, as the definition module docstring does. Two leftover "scope"s in the parsing page say "context", and the models page says "member" for an extra field. The example-drift test imports at module level. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): number the guide-pages fragment after upstream PR 4605 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): fence the module docstring examples as python so the API pages highlight them The examples in the definition, json_schema, pydantic and typed_json module docstrings were indented code blocks. An indented block names no language and pymdownx.highlight does not guess one, so the API reference rendered them as plain text. Each is now a ```python fence, and the example-drift test reads the definition docstring's fence. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): write the guide pages in short sentences and lists Long comma-chained sentences on the guide pages, README and landing page are split into short sentences, and enumerations become bulleted or numbered lists. models, validation-rules, consolidated-metadata, repair, extensions, json-schema and pydantic are restructured; context and parsing-validation keep their narrative voice and only have their long sentences split. No link, claim or code block changes. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…initions Export the rule that rejects a fill value string that is not standard-alphabet base64, so a data type defined outside the package can pass it as `fill_value_rules` or yield from it. `bytes` and `variable_length_bytes` now share it. The parsing and validation guide shows a data type that defines `fill_value_rules`, built with `union`. Rebased onto main as one commit; the earlier history carried the merged ZARR_PYTHON_CONTEXT branch. Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
force-pushed
the
claude/great-stonebraker-732ded
branch
from
November 9, 2026 12:31
20772a1 to
d9a80af
Compare
…s#4609 Assisted-by: ClaudeCode:claude-opus-5-5
…velopers#4521) * Remove zarr.tree and the Array.compressor properties Both were deprecated in 3.0.0 and were scheduled for removal in 3.2.0 by zarr-developers#3325. The rest of that PR shipped in 3.2.0 through zarr-developers#3900-zarr-developers#3903. Removed: - zarr.tree, zarr.api.synchronous.tree, zarr.api.asynchronous.tree. Use Group.tree / AsyncGroup.tree. - Array.compressor and AsyncArray.compressor. Use Array.compressors. The warning-only test for zarr.tree is gone, and the v2 compressor assertions in tests/test_v2.py now use `compressors`. The zarr.storage.default_compressor shim from the same zarr-developers#3325 list is left alone; zarr-developers#4505 removes it. --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…arr-developers#4504) * fix: reset the codec-pipeline thread pool after fork A forked child inherits codec_pipeline._pool pointing at threads that do not exist in the child, plus the parent's idle-semaphore permits. Submissions up to the permit count are consumed without spawning a thread, so small FusedCodecPipeline reads hang forever. Add an after_in_child hook that drops the inherited pool, size, and lock. The lock is recreated because it may have been held by a thread that no longer exists in the child. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: add changelog entry for zarr-developers#4483 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(codec_pipeline): pin pool size in the fork regression test Move the zarr-developers#4478 regression test next to the other FusedCodecPipeline pool tests and pin `codec_pipeline.max_workers` to 4 through the shared `_FUSED_POOL_CONFIG`. With the default, a 1-CPU host resolves `max_workers` to 1, never creates the pool, and passes the test without exercising the bug. The test now asserts that the parent created the pool before forking, and it runs on macOS, where the fork start method is available. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: barlowa124 <barlowa124@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: read only the needed rows of uncompressed chunks BytesCodec now implements partial decoding. An uncompressed chunk is stored in C order, so each row along its first axis is a contiguous run of bytes; for a selection that does not touch every row, the rows from the first to the last one it touches are fetched with a single range request and the selection is applied to them. The pipeline already uses partial decoding when the array-to-bytes codec supports it and there are no array-to-array or bytes-to-bytes codecs, so this applies to uncompressed arrays only. Reading 100 rows of a single-chunk 20 MB array now fetches the bytes of those rows instead of the whole chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix boolean masks and stores that ignore ranges in BytesCodec partial reads A boolean mask on the first axis, as from arr.oindex[mask], was trimmed to start at the first selected row but not to end at the last, so its length no longer matched the rows fetched and numpy raised an IndexError. The row window and the shifted selection are now computed together in _row_window, which trims the mask at both ends. This also gives mypy the narrowing it needs. FsspecStore over HTTP returns the whole object when a server ignores the Range header. The partial read then failed to reshape a chunk that the full read would have decoded, so a read that worked before this branch raised an error. A response longer than the requested range is now sliced to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add sync partial decode to BytesCodec and check the length of ranged reads Add _decode_partial_sync so FusedCodecPipeline reads only the needed rows of an uncompressed chunk, sharing the decode step with the async method. Decide what a store sent for a range request by its exact length: the requested range, the whole chunk, or everything from the range start. Any other length raises instead of returning wrong data. Run the partial read tests on both codec pipelines and add a changelog entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…4522) * Deprecate the ignored Zarr-Python 2 keyword arguments `synchronizer`, `chunk_store`, `cache_metadata`, `cache_attrs`, `object_codec`, `read_only` and `meta_array` are accepted by `create`, `group` and `open_group` (and by everything that wraps them, such as `zeros`) but have never done anything in Zarr-Python 3. Passing one used to emit a `ZarrRuntimeWarning` saying "not yet implemented", which read as a promise to implement it. Emit a `ZarrDeprecationWarning` instead, saying the argument is ignored and will be removed in a release after April 2027, and say the same in the docstrings and the v3 migration guide. Calls that do not pass these arguments are unaffected. First step of the plan for zarr-developers#3976. --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…ers#4612) Python shows a DeprecationWarning by default only when it is attributed to __main__, so warnings that pointed at zarr internals were hidden from users running scripts: - `ArrayV2Metadata.chunk_grid` is a cached_property, which adds a frame; stacklevel 2 pointed at functools. Now 3. - Setting `zarr.storage.default_compressor` used stacklevel 1, which pointed at zarr itself. Now 2.
…velopers#4610) * feat(zarr-metadata)!: write store bytes through a JSONWriter `to_key_value` takes a writer instead of an `indent` keyword. A `JSONWriter` owns how a document becomes JSON bytes: it holds the settings (`indent`, `sort_keys`, `ensure_ascii`, `trailing_newline`) and its `encode` method does the writing, so the separators, the bare non-finite tokens and the surrogate escapes are no longer split off in a private helper. `to_key_value` also accepts a mapping of the settings, `JSONWriterSchema`, so callers need not import the class: `to_key_value({"indent": 2})`. The defaults are those of `json.dumps`, so a bare `to_key_value()` writes the same bytes as before. A setting `json.dumps` would misread raises, and so does a mapping key that is not a setting. Rebased onto main as one commit, replacing the JSONWriterConfig design. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the changelog fragment for zarr-developers#4610 Assisted-by: ClaudeCode:claude-opus-5-5
Owner
Author
|
🤖 AI text below 🤖 Closed as superseded: this branch merged upstream as zarr-developers#4609. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Agent summary
Export
base64_fill_value_rulesfromzarr_metadata.v3.data_type.bytes, the rule that rejects a fill value string that is not standard-alphabet base64, so a data type defined outside the package can pass it asfill_value_rulesfor aBase64Bytesfill value, or yield from it in a rule of its own. The guide to parsing and validation now shows how to define a data type withfill_value_rules.Refs #425. The issue asks for a way to constrain a fill value beyond its JSON type.
fill_value_rulesalready does that, and both the reader and the pydantic field apply it. The consumer's definition only setfill_value=str, the guides never mentionedfill_value_rules, and the package's base64 rule was private. This PR fixes the last two.base64_fill_value_rules(inv3/data_type/_byte.py) fits thefill_value_rulesslot and checks only strings, so it also servesbytes, whose fill value may be written as byte values.bytesandvariable_length_bytesnow use it in place of their own copies.null_terminated_bytesandzarr_python_raw_bytesstill callbase64_problemsdirectly before their size checks.docs/parsing-validation.mdgains an example: anacme.packeddata type whose rule yields frombase64_fill_value_rulesand then checks the size its configuration sets. It adds the type withCORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(data_types=[PACKED])), and runs in the docs tests.json_schemamodule documents, so a schema for such a context accepts any string.The agent rebased this onto
mainas one commit: the branch had been stacked on theZARR_PYTHON_CONTEXTbranch, which merged upstream as zarr-developers#4583, and the docs example switched from the removedcontext.update(...)tounion.just checkinpackages/zarr-metadatapasses (3696 tests and the strict docs build).🤖 Generated with Claude Code