Repository navigation
Conversation
…s and narrowing helpers (zarr-developers#4638) * feat(zarr-metadata): export the v3 node types, spec default separators and narrowing helpers Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE`; the default separators `ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR`, `DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` and `V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR`; and the `TypeIs` helpers `is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` and `is_zarr_v3_node_type`. The v2 array reader, the v3 node reader and the chunk key encodings' canonicalization now use them in place of their own literals, which removes two casts from the v2 array reader. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): add the changelog fragment for the spec constants Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): name the changelog fragment after the upstream PR Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…ed chunk grid `ArrayV3Metadata` validated every codec against the stored chunk grid, even after a `TransposeCodec` filter had permuted the chunk shape. A sharding serializer whose inner chunk shape divided only the stored grid was accepted and then wrote data that read back wrong; one that divided the chunks it actually receives was rejected. Codecs gain a `resolve_chunk_grid` hook, the grid counterpart of `resolve_metadata`, which defaults to the identity. `TransposeCodec` permutes the grid's dimensions with it, moving a rectilinear grid's edge lists with their dimensions. Array metadata threads the grid through the codecs alongside the spec, and drops the second, unthreaded validate loop. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Fix array metadata validating each codec against the stored chunk grid even after a
TransposeCodecfilter had permuted the chunk shape. A sharding serializer whose inner chunk shape divided only the stored grid was accepted, then read back wrong data. A valid one was rejected. Codecs gain aresolve_chunk_gridmethod, the chunk-grid counterpart ofresolve_metadata, whichTransposeCodecuses to permute the grid.Summary
Repro on upstream main (95ec04a): shape (12, 40), chunks (6, 20),
filters=[TransposeCodec(order=(1, 0))].ShardingCodec(chunk_shape=(3, 10))was accepted. The sharding codec receives (20, 6) chunks, and writing then reading back returned different data (np.array_equalwas False).ShardingCodec(chunk_shape=(10, 3)), which is valid, was rejected with "Chunk edge length 6 in dimension 0 is not divisible by the shard's inner chunk size 10."Fix:
BaseCodec.resolve_chunk_grid(chunk_grid) -> ChunkGridMetadatais a new hook, the grid counterpart ofresolve_metadata. By default it returns the grid unchanged, which is correct for any codec that keeps a chunk's shape. The publicCodec.validatesignature does not change, so third-party codecs are unaffected.TransposeCodec.resolve_chunk_gridpermutes the grid's dimensions byorder, asresolve_metadatapermutes the shape.ArrayV3Metadata.__init__passes the grid through the codecs alongside the spec, so each codec is validated against the grid that the codecs before it leave._validate_metadatano longer runs its second codec validate loop, which ignored the earlier codecs. It also now runs before the codec loop, so an ndim mismatch between the grid and the shape is reported as such instead of surfacing inside a codec.docs/user-guide/extending.mdlists the new hook.Design: why a hook, and how rectilinear grids work
For a regular grid, the grid could be derived generically by passing the chunk shape through
resolve_metadata. Rectilinear grids rule that out. A transpose moves each dimension's edge list, and for a square array with different edge lists per dimension, nothing inresolve_metadata's output shows which dimension moved where. Only the codec knows its permutation, so the codec owns the mapping. Under a transpose, rectilinear edge lists (and bare-int dimensions) move with their dimensions.A third-party codec that changes chunk shape but does not override the hook keeps the behavior from before this PR, so it is no worse off.
The pipeline-level
CodecPipeline.validatemethods incodec_pipeline.pystill loop without passing the grid through, but nothing in the codebase calls them. They are out of scope here.Tests
test_sharding_after_transposeis one parametrized success test. It covers a regular grid, a regular grid transposed twice (a control), a 3-D non-self-inverse order, a rectilinear grid on a square array, and a rectilinear grid with a bare-int dimension. Each case writes and reads data back.test_sharding_after_transpose_indivisiblecovers the single error case, an inner chunk shape that divides the stored grid but not the transposed one, for regular and rectilinear grids.Stack notes
ShardingCodec.evolve_from_array_specand also drops_validate_metadata's validate loop. Expect a small conflict insrc/zarr/core/metadata/v3.pywhichever PR merges second. The resolution keeps the loop in__init__that passes the grid through.🤖 Generated with Claude Code