Repository navigation
Conversation
…y layer Declaring an array today eagerly stores its metadata, so chunks cannot be written first. ArrayIO and AsyncArrayIO bind an array metadata document to a store location without performing any IO on construction; storing metadata (store_metadata) and storing or reading chunks (setitem/getitem) are explicit method calls that run in whatever order the caller chooses, mirroring the array APIs of zarrs and zarrita. The normalization half of init_array is extracted into the pure build_array_metadata, which init_array and AsyncArrayIO.build share; init_array's behavior is unchanged. Refs zarr-developers#4217 Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
…ing open Every write now has an inverse, copying zarrs: store_metadata ↔ erase_metadata, chunk writes ↔ erase_chunk / erase_chunks, each succeeding when the target is already absent. open's zarr_format now defaults to None (read format 3, fall back to format 2), matching zarrs' open, instead of requiring format 3. Adversarial review against zarrs and TensorStore also found contract gaps now documented: the bound metadata is assumed, never verified (TensorStore's assume_metadata hazard); store_metadata keeps any previous array's chunks; a repaired document is stored before the first chunk write; and the module docstring no longer cites zarrita, whose create is eager. Assisted-by: ClaudeCode:claude-fable-5
… store A writer that declared an array independently (the distributed chunks-first pattern: one coordinator stores metadata, N workers build the same declaration) can assert its assumption before writing chunks. verify() reads the stored document, favoring format 3 when both exist, and raises naming each differing field — the explicit counterpart of TensorStore's spec-constraint validation on open. Assisted-by: ClaudeCode:claude-fable-5
The explicit-IO layer is the lower layer, so it moves into zarr.core.array as a self-contained class holding its own (metadata, store path, config, codec pipeline, chunk grid) state, with zarr.experimental.array_io re-exporting it. AsyncArray now holds one field, the wrapped AsyncArrayIO, exposes metadata/store_path/config/ codec_pipeline/_chunk_grid as properties over it, and delegates its metadata and selection IO to it; resize rebinds the wrapped state, so the two layers cannot drift. AsyncArrayIO defines value equality over the full binding, preserving AsyncArray's previous dataclass equality, and to_array() returns an AsyncArray sharing the same binding. The repair-before-chunk-write hook and the stored-document mark reset become module functions used by the one remaining implementation. Refs zarr-developers#4217 Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
ArrayIO and AsyncArrayIO move out of zarr.experimental into the zarr namespace, next to Array and AsyncArray, which are built on them: the wrapped layer is reachable as Array.io / AsyncArray.io, the inverse of to_array(). The zarr.experimental.array_io shim is removed and the layer is documented on the array API page. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
changed the title
zarr-python now depends on zarr-metadata (>=0.6.0): AsyncArrayIO holds one source — the document, a zarr_metadata model, when constructed by open; the resolved metadata when constructed by build or by AsyncArray — and derives the other form, the chunk grid, the chunk key encoding and the codec pipeline as caches. Document operations (store_metadata, erase_metadata, erase_chunk, erase_chunks, verify, equality) never resolve codec or data type names, so an array whose codecs this environment lacks still opens for document work and refuses only when the data path is built; a document zarr can only read as repaired is refused by the strict model read, with AsyncArray.open remaining the repairing reader. Grid derivation goes through the same grid-metadata objects ChunkGrid.from_metadata dispatches on, so both derivations lay out chunks identically, and model JSON is normalized through JSON text because the model returns read-only forms. Resolved-seeded instances keep deriving their grid, chunk keys and equality from the resolved metadata: zarr-python writes some documents the strict model refuses (Zarr format 2 generic time units), and the array API's behavior must not depend on the model accepting them. Refs zarr-developers#4217 Assisted-by: ClaudeCode:claude-fable-5
…ce once Binding is pure, so build stops being awaitable: it takes a Store or a StorePath and binds with StorePath construction, never make_store_path, whose coercion of path and URL strings opens stores — the IO the method promises not to do. Stores open themselves on their first IO, so an unopened store binds safely. The declaration keyword surface is declared once, as the ArrayDeclarationKwargs TypedDict that both build methods unpack, instead of being copied into each; zarr_format on build no longer admits None, whose config-default meaning collided with open's auto-detect meaning one method away. The constructors return Self, and build_array_metadata's fill_value drops a redundant None from its type. Assisted-by: ClaudeCode:claude-fable-5
…outine Nothing hides the v2 / v3 difference any more: build_array_metadata is replaced by module-level build_array_metadata_v2 and build_array_metadata_v3, each taking the actual fields of its document (.zarray plus .zattrs, and zarr.json), with the bodies lifted from the AsyncArray._create_metadata_* staticmethods, which now delegate (the module-level home zarr-developers#4175 asks for). init_array regains its normalization and calls the per-format builders. ArrayIO and AsyncArrayIO declare with build_v2 and build_v3 over per-format TypedDicts, so a format mistake — shards on a v2 array, order on a v3 one — is inexpressible instead of guarded, and build_v3's chunk_grid accepts the grid metadata, its JSON, or a chunk shape declaring a regular grid, as ArrayBuilder does in zarrs. Refs zarr-developers#4175 Assisted-by: ClaudeCode:claude-fable-5
d-v-b
commented
Oct 31, 2026
| ) | ||
|
|
||
| @property | ||
| def document(self) -> ArrayDocumentModel: |
Owner
Author
There was a problem hiding this comment.
rename to metadata_document
Owner
Author
There was a problem hiding this comment.
🤖 AI text below 🤖
Superseded by the follow-up naming: the property is metadata (the zarr-metadata model) and the legacy resolved view is metadata_legacy (8956b96).
d-v-b
commented
Oct 31, 2026
|
|
||
| def __init__( | ||
| self, | ||
| document: ArrayDocumentModel | ArrayMetadata | ArrayMetadataDict, |
Owner
Author
There was a problem hiding this comment.
the bare name "document" is confusing. imo we should just call it metadata.
Owner
Author
…ta_legacy ArrayIO's metadata is the zarr_metadata model, full stop: the property is named metadata, the resolved ArrayV2Metadata / ArrayV3Metadata view — which the data path runs on and AsyncArray.metadata still returns — is metadata_legacy, and the term document leaves the surface. The with_metadata, with_store and with_config methods return new bindings with one part replaced, with with_config merging partial configuration the way AsyncArray.with_config does. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
…does The codec pipeline implementation is captured when the binding is made — the capture discipline AsyncArray has always had — through a new pipeline_class parameter on create_codec_pipeline, while the pipeline itself still builds on first use, so metadata-only work resolves nothing. build_v2's order defaults from the global array.order instead of a static C. Nothing about ArrayIO requires new configuration concepts; scoping resolution per binding stays in the follow-up that deprecates the global state. Assisted-by: ClaudeCode:claude-fable-5
The zarr.json field is data_type, so build_array_metadata_v3 and the build_v3 declaration take data_type; the v2 builder keeps dtype, which is the .zarray field's own name. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
…data Addresses the roborev review of this branch (job 1359). with_metadata, with_store and with_config copy a legacy source object, so the _stored_document repair mark and in-place attribute updates never leak between bindings, and they carry the captured pipeline implementation instead of re-reading the global configuration at call time; AsyncArray.with_config routes through the binding and inherits all of that. Storing a v2 model without attributes now deletes a stale .zattrs, matching the unconditional-replacement contract, and the model branch honors json_indent. erase_chunks reads async.concurrency with an explicit None check instead of a falsy or-default. Behavioral tests cover array-to-binding state propagation, derivative independence, and the stale-.zattrs case. Assisted-by: ClaudeCode:claude-fable-5
`copy.copy` on a legacy metadata object shared its `attributes` dict, so an in-place attribute update still reached bindings derived before it. Derived sources now go through `_copied_legacy`, which copies the dict and keeps the repair mark. `_derive` fills the fields of a new binding directly instead of re-running `__init__` (which re-read the global pipeline choice), and `AsyncArray.with_config` builds the derived pipeline eagerly and constructs through `type(self)`, keeping subclass identity and warning timing. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… on metadata Whether a stored array document needs a repair that moves chunks is a fact about the store, so it now lives in `zarr.core.metadata.stored`: a cache per store object, keyed by array path and Zarr format, filled by metadata reads (once the document has been read as metadata) and writes, and consulted before chunk writes, which read the store on a miss. Consolidated copies of such documents are recorded apart, so a group write keeps them as stored. This removes the `_stored_document` attribute from `ArrayV2Metadata` and `ArrayV3Metadata`, and every place that had to carry or clear it: derived bindings, re-resolution and resize no longer manage store state. `mark_repaired` becomes `warn_repairs`, and each `from_dict` is a private `_read_stored` reader plus its warning, so the cache can read stored documents without the rectilinear flag that gates storing them. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…quires zarr-metadata 0.6.0 requires typing_extensions>=4.16, so zarr's effective minimum was already 4.16 while it declared >=4.14, and the min_deps test environment's typing_extensions==4.14.* pin could not resolve. Declare the real floor and pin min_deps to 4.16.*. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A store's cache does not pickle with the store, so every unpickled binding (each task of a distributed write) read its metadata document again before its first chunk write. A pickled AsyncArrayIO now carries what its store's cache knew of its own document, restored into the new store object's cache unless that one already knows it, as the removed mark used to pickle with the metadata. The StoredDocuments docstring now says that a chunk write's own read is recorded as read. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`__init__` and `_derive` each listed every field of an AsyncArrayIO; both now go through `_bind`, so a field added later cannot be missed by one of them. `_copied_legacy` drops a guard that never fired and says what its copy isolates: top-level attribute updates. Tests cover with_store and with_metadata from a legacy source, a derived binding keeping its captured pipeline while the global configuration names one that cannot load, and StoredDocuments.restore keeping what the cache learned itself. The installation guide gives the new typing_extensions floor. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds it with_metadata copied a legacy object without the reading parse_array_metadata gives it, so ArrayV2Metadata with a chunk size of 0 bound unrepaired there while the constructor bound it repaired. A test binds such metadata through the constructor, build_v2 and with_metadata. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repair check before a chunk write is only as fresh as the store object's cache: once another binding of that store object has read or stored a newer document, an array opened before then writes its chunks under its own metadata without the check, as any stale array handle does. This is the accepted behavior of the per-store cache; the removed metadata mark had re-read the store for such handles. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ject codecs A v2 binding read as a zarr-metadata model copied only the top level of the model's read-only attributes into its legacy metadata, so nested lists and objects stayed tuples and mapping proxies and every metadata write through it (update_attributes, resize, append) failed to encode. The attributes now come from the model's own `.zattrs` document. build_array_metadata_v2 (and build_v2) accepted a variable-length string data type with no object codec, storing metadata nothing can read; it now makes the check create_array makes, shared as `_check_object_codec_v2`. store_metadata's docstring says what it replaces: the documents of its own format, leaving the other format's, a group's and created parents in place. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AsyncArray's only dataclass field had become the wrapped binding, so dataclasses.fields/asdict/replace, the generated equality and hashing, the pickled state and dask tokens (dask tokenizes dataclasses by their fields) all changed, and copy.copy shared one binding. AsyncArray now declares the fields it always had (metadata, store_path, codec_pipeline, _chunk_grid, config), each a property over the binding that object.__setattr__ can still set, and pickles as that field state, so pickles load across versions in both directions and copies are independent. _save_metadata keeps its signature, and with_config rebuilds the array as it always did, re-reading the global pipeline choice. A chunk write no longer reads a document the store's cache has not seen: unknown means nothing is known to repair, as unmarked metadata always did, so writes through consolidated members, code-built arrays and their unpickled copies make no new metadata reads. A pickled binding carries a consolidated copy's entry as well as its document's. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…natures AsyncArrayIO and ArrayIO read a metadata dict as a zarr-metadata model when it is valid JSON metadata, as zarrista's from_metadata does, so an extension zarr-python lacks fails only when chunks are read or written; metadata built in code, or an invalid document zarr reads as repaired, is still read into the legacy classes when bound. AsyncArray keeps reading dicts the legacy way. erase_chunk takes grid coordinates positionally, as any sequence, and setitem takes its prototype by keyword, matching zarrista before either signature is released. AsyncArray's pickled state carries the pipeline class its binding chose, which may differ from its pipeline instance's class. The object-codec check applies to build_array_metadata_v2 only, so zarr.create(zarr_format=2) behaves as before. Also: drop an unpickling shim for a state no release wrote, a dead read guard, and a duplicate attribute encode. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…els lazily A pickled or deep-copied consolidated group lost what its store's cache knew of its members, so a group write through the copy re-encoded a member read from a repaired copy instead of keeping it as stored (and raised for a rectilinear one). AsyncGroup now carries those entries in its pickled state. An AsyncArray built from a stored document given as a dict records it, so its first chunk write stores the repair, as before. The cache records a valid document only where it replaces one that needs a repair or a consolidated copy that does, so it no longer grows with every array a store object sees; group writes are not re-parsed to record them, and zarr_metadata is imported when a binding first holds a model, so `import zarr` costs what it did. A legacy-sourced binding derives its model on each access, so an in-place change through a handle sharing its metadata is never stale. ArrayIO.open opens a path writable without creating it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…antics A metadata dict with storage transformers bound as a model and failed only at the first chunk operation; it is now rejected when bound, as AsyncArray always rejected it. The cache's `knows` and the repair check's docstring say that a valid document is recorded only where it replaces a broken one, and a test pins that a member read valid and then from a broken consolidated copy is kept as the copy stored, as before. Tests also cover a pickled subgroup of a consolidated parent, a binding derived from an unpickled array building the pipeline class it chose, and `import zarr` leaving zarr_metadata unimported. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipelines ArrayIO and AsyncArrayIO type their metadata argument to take zarr_metadata's JSON types (what zarrista's arrays hold), and with_metadata reads a metadata dict as the constructor does instead of binding the dict itself. A pickled AsyncArrayIO no longer carries its built codec pipeline, which is bound to the pickling process's store object and may not pickle; it is rebuilt on first use. The class docstring words name resolution in the data path as what today's pipelines do. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fields erase_metadata deleted the array documents of its format at its location even when they were a group's (a v3 group's zarr.json, a v2 group's .zattrs); it now deletes only an array's, and storing an attribute-less v2 model leaves a group's .zattrs in place. The AsyncArrayIO constructor copies a legacy object it is given, as with_metadata and the derivations do, while AsyncArray keeps binding the object itself through _from_resolved. A metadata dict whose reading moves chunks binds as repaired legacy metadata and is recorded as the stored document, also where the model accepts it (a v2 chunk size of 0). verify names the fields that differ as the model compares them. build_array_metadata_v3 keeps an empty dimension_names list (the creation paths still read it as none, as they always have), erase_chunk takes integer coordinates only, and binding equality no longer depends on what has been resolved. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… before zarr 3.4.1 and 3.4.2 released the repair mark on metadata objects (`_stored_document`), so pickles cross versions: an array pickled by them now has its mark read into the store's cache, a consolidated group's members likewise, and an array pickled here marks a copy of its metadata the same way for those versions to read. A v3 group write no longer encodes its consolidated metadata twice (30% of every write on a large consolidated group), and the check for whether a document's repair moves chunks returns early for documents whose regular chunk sizes are all positive, so reads of consolidated metadata cost what they did. AsyncArray and AsyncArrayIO raise TypeError for metadata that is not a mapping, as AsyncArray always did. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…documents ArrayIO.open and a model given to the constructor bound metadata declaring storage transformers and failed only at the first chunk operation; every model is now checked when bound, as dicts and AsyncArray are. erase_metadata no longer parses documents before deleting them, so a corrupt or node_type-less array document is removed rather than raising or surviving; only a group's documents are left (a v2 `.zattrs` beside a `.zgroup`, a v3 group's `zarr.json`). Pickles no longer carry a stale 3.4.x mark, and a consolidated group pickled here marks its members for 3.4.1 and 3.4.2 to read. Binding equality is always the models' canonical equality, the rectilinear grid skips the repair scan it never needed, AsyncArrayIO names the type it was given when it cannot bind it, and with_metadata takes any mapping. Tests pin these and check may_move_chunks against the repairs. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… chunk size 0 on strict reads - verify builds both sides as plain JSON, so nested attributes (read-only mapping proxies in the model) no longer raise TypeError for v2 arrays. - The strict read (`open`, `verify`) rejects a v2 chunk size zarr reads as repaired, as it does for v3, though the v2 model accepts it. - with_metadata raises TypeError for anything that is not array metadata. Assisted-by: ClaudeCode:claude-opus-5-5
…ubclass state in pickles - A metadata write records its documents in the store's cache only when the cache knows the path or the written metadata might move chunks, judged from its chunk sizes; writes of large attributes cost what they did on main. store_documents takes the written array metadata (default: none, a group's), so its two-argument form works again. - AsyncArray pickles keep a subclass's own instance attributes, and a shallow copy shares the metadata object again, also for a repaired array. - Two threads first touching one store object share one cache. - zarr.core.group re-exports Store and set_or_delete, as 3.4.2 did. Assisted-by: ClaudeCode:claude-opus-5-5
…model's chunks - A change to an AsyncArray's state (resize, append, attribute updates, a field set with object.__setattr__) gives it a binding of its own, so an array copied by its __dict__, or a binding taken from `io` before the change, keeps the state it had, as when the fields were its only state. - An AsyncArray over a model-backed binding makes the metadata it hands out the binding's source, so the binding's model never disagrees with it. - A zarr_metadata v2 model bound through the constructor or with_metadata is checked as the strict read checks it (a chunk size of 0 is rejected), and a model with storage transformers is rejected by with_metadata too; a tuple-valued mapping with chunk size 0 binds as repaired, as AsyncArray reads it. - Bindings of different classes that compare equal hash equally. Assisted-by: ClaudeCode:claude-opus-5-5
- An array made from a binding (`to_array`) works on a copy of it, so nothing done through the array (an attribute update, a resize) changes the binding it came from, and reading the array no longer replaces the binding's model with its legacy round trip, which lost NaN payloads, codec configuration and v2 float fill values. - A model-backed binding whose resolved metadata an array changed in place (attribute updates through copies sharing it) reports those attributes in its model, and stores them. - Copies and pickles of an array keep the class of its binding. - The dataclass fields of AsyncArray are no longer attributes of the class, so a @DataClass subclass may annotate one again or add a field without a default, as before. - The strict v2 read reports the model's problems together with its chunk sizes; binding is the one place a model is checked; a metadata mapping holding tuples is read once, as its JSON; parse_array_metadata's TypeError names what it got, and the storage-transformer error has its space back. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…ough rebuilds - Every way a binding is made or changed goes through `_bind`, including the in-place rebind of a resize or attribute update, so a binding subclass that derives state of its own there (a native handle) never goes stale. - An array over a model binding keeps the model when rebuilt by `with_config`, pickling or a deep copy, so its binding still verifies against an unchanged store and stores the document as it was (NaN payloads, codec configuration); `with_config` also keeps the binding's class. - Binding equality no longer raises when attributes were changed in place to values JSON cannot hold, and reading the model names such a value. Assisted-by: ClaudeCode:claude-opus-5-5
…NG on the class - AsyncArray pickles and copies carry the values of slots a subclass declares, and a subclass that defines its own reduction (`__reduce_ex__`, `__reduce__`, `__getstate__`, `__setstate__`) is copied its way. - The dataclass fields read as `dataclasses.MISSING` on the class instead of raising, so a loop over `dir(AsyncArray)` works while a `@dataclass` subclass still finds no default. Assisted-by: ClaudeCode:claude-opus-5-5
…els portably - AsyncArray.with_config keeps the metadata a subclass __init__ built, binding the model only when the constructor bound this array's own metadata. - Pickles carry slots as the default reduction does (the instance dict and the slots' values, name-mangled ones included), so a slotted subclass's pickle loads in either version, and carry a model binding's documents as bytes, so a zarr without zarr-metadata loads it. - Unpickling a binding, and setting an array's store_path or config, go through `_bind`; the docstring says attributes alone change in place. Assisted-by: ClaudeCode:claude-opus-5-5
…each problem once - AsyncArray pickles carry the binding's own state (pipeline class, binding class, a model's documents, the store cache's entry) under one key, so a subclass attribute of any other name pickles as it always did. - The strict v2 read reports a chunk size the model already rejected once. - A model binding compares changed attributes as JSON text and keeps the model it made for them, so NaN attributes are stable and reads are cheap; equality without JSON attributes compares documents without them, then the resolved attributes. - A subclass that defines its own reduction is copied without private `copy` helpers; `__getstate__` is annotated with the tuple it may return; the docstring says a binding whose derived state does not pickle leaves it out. Assisted-by: ClaudeCode:claude-opus-5-5
…ubclass reductions - A pickled array's model is bound again only when the pickle still holds the metadata it was pickled with: a zarr that keeps the binding's state but changes the metadata (3.4.2 resizing an unpickled array) no longer brings back a stale model. - A subclass whose own reduction carries a state of its own, with no `__setstate__`, copies and pickles as the default protocol would. - Attributes changed in place are written in their own order; the strict v2 read reports a chunk size of 0 beside the model's other problems. Assisted-by: ClaudeCode:claude-opus-5-5
…d chunk size once - A model binding compares attributes changed in place as JSON makes them (keys as strings), so attributes holding an int key no longer raise. - The strict v2 read adds only an int chunk size of 0 to the model's problems; `False` and `0.0`, which the model rejects itself, are reported once. Assisted-by: ClaudeCode:claude-opus-5-5
`AsyncArray.with_config` carries the pipeline its constructor built into the rebound binding only when that binding runs the same pipeline class, so a binding subclass whose `_bind` chooses its own pipeline (a backend) runs it. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A binding over a model resolves its legacy metadata lazily. Threads that resolved it at once each kept their own object, so an attribute update made through one was lost when another stored its resolution afterwards. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bute order AsyncArray.__setstate__ took any state holding a `metadata` entry for its own field state, and raised on a subclass reduction's `(None, slots)` state; it now recognizes its own state by both `metadata` and `codec_pipeline`. AsyncArrayIO's pickle carries its store-cache entry under one key of its own, so a subclass attribute of the same name survives, and leaves out the attributes' derived model. Attributes reordered in place now count as changed, so the written order reaches the model. _rebound_like says a binding's pipeline class wins over one a subclass __init__ assigned, and a test pins it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery field AsyncArray.__setstate__ took a state with `metadata` and `codec_pipeline` for its own field state and then raised KeyError on the fields it lacked, so a subclass reduction holding just those two could not load or copy. It now requires all five fields. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Added
zarr.ArrayIOandzarr.AsyncArrayIO, an explicit-IO layer over Zarr arrays. Constructing one performs no IO against storage: storing metadata (store_metadata), storing or reading chunks (setitem/getitem), and their erasing inverses (erase_metadata,erase_chunk,erase_chunks) are explicit method calls that run in whatever order the caller chooses — so chunks can be written before metadata, as in the array API ofzarrs.openauto-detects the Zarr format by default, aszarrsdoes, andverifyreads the stored metadata and raises, naming each differing field, when it does not match the bound metadata.The binding holds the metadata document as a
zarr-metadatamodel — zarr-python now depends onzarr-metadata>=0.6.0, its first sibling-package dependency — reachable asArrayIO.document. Document operations (store_metadata,erase_metadata,erase_chunk,erase_chunks,verify, equality) never resolve codec or data type names: an array whose codecs this environment lacks still opens for document work and refuses only when chunks are read or written. The strict model read refuses a stored document zarr can only read as repaired (AsyncArray.openremains the repairing reader), and instances built from resolved metadata — everyAsyncArray— keep deriving their chunk grid, chunk keys and equality from the resolved form, because zarr-python writes some documents the strict model refuses (Zarr format 2 generic time units).AsyncArrayis now an array-like layer over this one: it holds a single wrappedAsyncArrayIO, exposesmetadata/store_path/config/codec_pipelineas properties over it, and delegates its metadata and selection IO to it, withresizerebinding the wrapped state so the two layers cannot drift.AsyncArrayIO.to_array()returns anAsyncArraysharing the same binding.AsyncArrayIOdefines value equality over the full binding, preservingAsyncArray's previous dataclass equality.The normalization half of
init_arrayis extracted into the purebuild_array_metadata, whichinit_arrayandAsyncArrayIO.buildshare;init_array's behavior is unchanged.Merge posture: this PR introduces no new configuration concepts.
ArrayIOmeets the global configuration exactly asArrayalways has — the codec pipeline implementation is captured when a binding is made, the runtime configuration defaults fromzarr.config,build_v2'sorderdefaults fromarray.order, andbuild_v3's default codecs come from thev3_default_*keys. Scoping resolution per binding (and the path to deprecating the global state) is the follow-up, #420.Design notes:
build_v2andbuild_v3each take the actual fields of their format's document (per-formatArrayV2DeclarationKwargs/ArrayV3DeclarationKwargsTypedDicts), so a format mistake —shardson a v2 array,orderon a v3 one — is inexpressible instead of guarded. The documents themselves are built by module-levelbuild_array_metadata_v2/build_array_metadata_v3, lifted from theAsyncArray._create_metadata_*staticmethods (the module-level home misplaced metadata creation methods on AsyncArray class zarr-developers/zarr-python#4175 asks for). The declaring constructors are synchronous: binding is pure, so they take aStoreorStorePath(coercing path and URL strings opens stores, which stayscreate_array's job), and stores open themselves on their first IO.build_v3'schunk_gridaccepts the grid metadata, its JSON, or a chunk shape declaring a regular grid, asArrayBuilderdoes inzarrs.store_metadata↔erase_metadata, chunk writes ↔erase_chunk/erase_chunks), each succeeding when the target is already absent, copying zarrs.store_metadata()writes unconditionally, likestore_metadatainzarrs; guarded creation (raising when a node exists) remainscreate_array's job, and chunks of a previous array at the location are kept —erase_chunksis the broom.assume_metadatahazard); this is documented on the class.verifyis the explicit check.zarr.core.arrayand are exported from thezarrnamespace, next to the classes built on them; the wrapped layer is reachable asArray.io/AsyncArray.io, the inverse ofto_array().Issues this layer addresses or advances: zarr-developers#4217 (the 2026-10-29 comment: avoiding implicit IO, writing chunks before metadata — the
Array-over-engine layering that issue describes now exists at the array layer), zarr-developers#1435 (idempotent, non-deleting parallel creation), zarr-developers#4175 (build_array_metadatamoves metadata creation offAsyncArray), zarr-developers#4399 / zarr-developers#1754 (storage-free declaration with caller-scheduled IO), zarr-developers#3017 (metadata IO as explicit free-standing operations), and zarr-developers#4027 / zarr-developers#3782 (theArray/AsyncArrayrelationship: the syncArrayIOstill routes throughsync(), so the event-loop issues — zarr-developers#2878, zarr-developers#3487, zarr-developers#3859 — need a sync engine under zarr-developers#4217 before they close).Stack
Base of the stack. #420 (Context scoping) stacks on this branch.
🤖 Generated with Claude Code