Skip to content

fix(metadata): validate codecs after a transpose against the transposed chunk grid - #449

Open
d-v-b wants to merge 4 commits into
mainfrom
fix/transpose-chunk-grid-validation
Open

d-v-b wants to merge 4 commits into
mainfrom
fix/transpose-chunk-grid-validation

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Fix array metadata validating each codec against the stored chunk grid even after a TransposeCodec filter had permuted the chunk shape. A sharding serializer whose inner chunk shape divided only the stored grid was accepted, then read back wrong data. A valid one was rejected. Codecs gain a resolve_chunk_grid method, the chunk-grid counterpart of resolve_metadata, which TransposeCodec uses to permute the grid.

Summary

Repro on upstream main (95ec04a): shape (12, 40), chunks (6, 20), filters=[TransposeCodec(order=(1, 0))].

  • ShardingCodec(chunk_shape=(3, 10)) was accepted. The sharding codec receives (20, 6) chunks, and writing then reading back returned different data (np.array_equal was False).
  • ShardingCodec(chunk_shape=(10, 3)), which is valid, was rejected with "Chunk edge length 6 in dimension 0 is not divisible by the shard's inner chunk size 10."

Fix:

  • BaseCodec.resolve_chunk_grid(chunk_grid) -> ChunkGridMetadata is a new hook, the grid counterpart of resolve_metadata. By default it returns the grid unchanged, which is correct for any codec that keeps a chunk's shape. The public Codec.validate signature does not change, so third-party codecs are unaffected.
  • TransposeCodec.resolve_chunk_grid permutes the grid's dimensions by order, as resolve_metadata permutes the shape.
  • ArrayV3Metadata.__init__ passes the grid through the codecs alongside the spec, so each codec is validated against the grid that the codecs before it leave.
  • _validate_metadata no longer runs its second codec validate loop, which ignored the earlier codecs. It also now runs before the codec loop, so an ndim mismatch between the grid and the shape is reported as such instead of surfacing inside a codec.
  • docs/user-guide/extending.md lists the new hook.
Design: why a hook, and how rectilinear grids work

For a regular grid, the grid could be derived generically by passing the chunk shape through resolve_metadata. Rectilinear grids rule that out. A transpose moves each dimension's edge list, and for a square array with different edge lists per dimension, nothing in resolve_metadata's output shows which dimension moved where. Only the codec knows its permutation, so the codec owns the mapping. Under a transpose, rectilinear edge lists (and bare-int dimensions) move with their dimensions.

A third-party codec that changes chunk shape but does not override the hook keeps the behavior from before this PR, so it is no worse off.

The pipeline-level CodecPipeline.validate methods in codec_pipeline.py still loop without passing the grid through, but nothing in the codebase calls them. They are out of scope here.

Tests
  • test_sharding_after_transpose is one parametrized success test. It covers a regular grid, a regular grid transposed twice (a control), a 3-D non-self-inverse order, a rectilinear grid on a square array, and a rectilinear grid with a bare-int dimension. Each case writes and reads data back.
  • test_sharding_after_transpose_indivisible covers the single error case, an inner chunk shape that divides the stored grid but not the transposed one, for regular and rectilinear grids.
  • Every new case except the twice-transposed control fails against upstream main's source. The full suite passes: 13258 passed, 1251 skipped, 9 xfailed. pre-commit passes, including mypy.
Stack notes

🤖 Generated with Claude Code

d-v-b and others added 4 commits November 16, 2026 13:48
…s and narrowing helpers (zarr-developers#4638)

* feat(zarr-metadata): export the v3 node types, spec default separators and narrowing helpers

Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE`; the default separators
`ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR`,
`DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` and
`V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR`; and the `TypeIs` helpers
`is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` and
`is_zarr_v3_node_type`. The v2 array reader, the v3 node reader and the
chunk key encodings' canonicalization now use them in place of their own
literals, which removes two casts from the v2 array reader.

Assisted-by: ClaudeCode:claude-opus-5-5

* docs(zarr-metadata): add the changelog fragment for the spec constants

Assisted-by: ClaudeCode:claude-opus-5-5

* docs(zarr-metadata): name the changelog fragment after the upstream PR

Assisted-by: ClaudeCode:claude-opus-5-5
…ed chunk grid

`ArrayV3Metadata` validated every codec against the stored chunk grid, even
after a `TransposeCodec` filter had permuted the chunk shape. A sharding
serializer whose inner chunk shape divided only the stored grid was accepted
and then wrote data that read back wrong; one that divided the chunks it
actually receives was rejected.

Codecs gain a `resolve_chunk_grid` hook, the grid counterpart of
`resolve_metadata`, which defaults to the identity. `TransposeCodec` permutes
the grid's dimensions with it, moving a rectilinear grid's edge lists with
their dimensions. Array metadata threads the grid through the codecs alongside
the spec, and drops the second, unthreaded validate loop.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant