Skip to content

feat(zarr-metadata)!: define codecs by kind and read each data type's number class - #4628

Merged
d-v-b merged 13 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-codec-data-type-support
Nov 12, 2026
Merged

d-v-b merged 13 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-codec-data-type-support

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR ensures that we don't allow invalid codec x dtype combinations, like allowing scale_offset to take a string dtype. It does so by making the codec definitions more expressive, and introducing classes for data types like integral, floating point, etc.

🤖 AI text below 🤖

Breaking: a v3 codec is defined with the class of its kind, ArrayArrayCodecDefinition, ArrayBytesCodecDefinition or BytesBytesCodecDefinition, instead of CodecDefinition(kind=...). The kind is now the class's kind property, and CodecDefinition is their abstract base, which a scope and resolve still use to name the kind. A codec that receives an array, an ArrayCodecDefinition, has chunk_rules and pipelines, an array -> array codec also has transition, and a bytes -> bytes codec, which receives bytes, has none of them, so building one with any of them is a TypeError from the class itself.

A v3 data type definition declares what kind of numbers its values are with number_class, a NumberClass: "integral", "floating_point", "complex" or "non_numeric". cast_value and scale_offset read it to decide which data types they accept, where each kept a list of data type names, so a data type defined elsewhere is judged by what it declares: both codecs refuse one that declares complex or non-numeric values, and cast_value refuses to wrap to one that declares floating-point values. A data type that declares no number class is accepted by both, as before. A data type defined elsewhere under one of the float names the cast_value spec lists, such as bfloat16, was refused as a wrap target by its name, and is now refused once it declares "floating_point".

Breaking: cast_value and scale_offset refuse the data types their specs do not list. Each spec names the integer and floating-point types as "the following data types defined in this repository" that it supports, and that repository also defines the numpy time types and the complex types. So both codecs now refuse numpy.datetime64 and numpy.timedelta64, and cast_value refuses them as the data type it casts to, and scale_offset now refuses complex64 and complex128. All of these were accepted. The time types' own spec says they are "compatible with any codec that supports arrays of signed 64-bit integers", which the two codecs' specs do not bear out. zarr_metadata.testing.strategies no longer draws these combinations.

Agent summary

Three commits, each of which passes just check on its own.

1. One class per codec kind (3348a561d)

  • CodecDefinition was built with kind="array_array" | "array_bytes" | "bytes_bytes", and a table (_UNASKED) refused the members a codec of a kind is never asked: the chunk rules, transition and pipelines of a bytes -> bytes codec, and the transition of an array -> bytes codec.
  • Each kind is now a class. ArrayCodecDefinition holds chunk_rules and pipelines, ArrayArrayCodecDefinition adds transition, and BytesBytesCodecDefinition has none of them. CodecDefinition is an abstract base (ABC, with kind an abstract property), and it is still the kind that scopes and resolve name.
  • The pipeline and the strategies dispatch on the class instead of comparing kind strings, so pyright sees which codecs have which members.
  • A definition's repr shows its class, as in BytesBytesCodecDefinition(name='gzip'), so the error messages that include it changed.

2. number_class on data type definitions (8d8e730ec)

  • cast_value and scale_offset decided which data types they accept from three sets holding 25 data type names, 11 of them float types this package does not define, kept so that wrap could refuse them.
  • A data type now declares a property of itself, as it does storage, and the two codecs' own rules read it. Their specs state the criterion the same way ("data types that model real numbers", "an integral type").
  • Neither a codec nor a data type names the other, so an extension codec can read the class of a core data type, and an extension data type is judged by what it declares.
  • None, the default, means unknown, and both codecs accept it, as the bytes codec accepts an unknown storage.

3. Verdicts that follow the codec specs (28e0bfd0a)

  • The time types declare "non_numeric", and scale_offset refuses the same classes as cast_value, so one set of refused classes serves both codecs.
  • scale_offset's message is now "expected a chunk of an integer or floating-point data type, got ...", since "a data type with arithmetic" would be wrong for complex numbers.
Design notes
  • Rejected: pairwise opt-in, as zarrs does it. A first version gave data types a supported_codecs set of codec names and codecs a requires_data_type_support flag, modeled on zarrs's define_data_type_support!. In zarrs the per-codec trait carries behavior and properties (cast_value_is_integral()), and either crate can add a pair. As a name list it lost both: cast_value still needed a list of float names for wrap, and an extension codec could not accept core int8 without redefining it. The agent replaced it after a review of the design.
  • An inference about complex numbers. The scale_offset spec's list covers only the data types of its own repository, and it leaves out all 13 complex types defined there. The agent read that as "no complex numbers", so a data type defined elsewhere that declares "complex" is refused by scale_offset too. The alternative is to refuse only complex64 and complex128 by name.
  • zarrs. zarrs registers only the integer and float types for cast_value, so it gives the same verdict for the time types. It has no scale_offset. The agent read the zarrs source at db0f278 and did not run it, and did not check tensorstore.
  • Not raised upstream. The time types' spec in zarr-extensions still says they are compatible with any codec that supports arrays of signed 64-bit integers, which the two codec specs contradict.
Testing
  • tests/v3/test_number_class.py: a table pins the number class of each of the 24 data types this package defines. A second table runs an extension data type of each class, and one that declares none, through both codecs, as input and as cast_value's target with and without wrap, and checks where each problem is reported and its message. One error test covers a number_class that is not one of the values.
  • tests/v3/test_pipelines.py: complex and both time types joined the table of data types each arithmetic codec refuses. The two cases that pinned the old acceptance are removed.
  • tests/testing/test_strategies.py: the test that reached complex scale_offset values is removed, and the tests of complex scales and time-type cast targets lost those cases.
  • The tests of codec kinds build each kind by its class, and check that the base class refuses to be built and that a kind refuses a member it does not have.
  • Locally, just check passes on the head commit: ruff, pyright (0 errors), the suite on Python 3.11 and on the newest supported version, and mkdocs build --strict.
Stack and bookkeeping

🤖 Generated with Claude Code

d-v-b added 10 commits November 12, 2026 14:22
Every extension module held its types (name constants, `Literal` names,
configuration and object `TypedDict`s, metadata aliases) beside its
definition. Codec modules imported data type modules at run time, so a
data type that needed a codec's types could import them only under
`TYPE_CHECKING`.

The types now live in modules that hold types only and import no
definition: one module per kind under `v3/_types/` (codec, data_type,
chunk_grid, chunk_key_encoding, consolidated) and `v2/_types/` (codec,
data_type). Their packages' `__init__`s hold no definitions, so importing
a types module never runs one. `EmptyConfiguration` moves to `_common`,
and `ZarrV2DataTypeDeclaration` to `v2/_declaration_json`.

Rules and other functions stay beside their definition, which imports
its types from the types module. Each public extension module re-exports
its types, so every public import path is unchanged. The codec rules
that name data types (`_arithmetic`, `vlen_utf8`, `vlen_bytes`,
`sharding_indexed`) import those names from the types module.

Assisted-by: ClaudeCode:claude-opus-5-5
Each types-only module is imported in a fresh interpreter in which the
`zarr_metadata`, `zarr_metadata.v2` and `zarr_metadata.v3` packages
(whose `__init__`s import every definition) are empty stand-ins. The
test asserts that no module holding a `Definition` ran. Every other
package `__init__` runs for real, so a types module placed in a package
that exports definitions fails. A control test checks that the probe
does see a definition module when an extension module is imported.

Assisted-by: ClaudeCode:claude-opus-5-5
Brings in zarr-developers#4621, which added `array_bytes_codec_default` and
`fill_value_default` to the data type definitions. Its three
`TYPE_CHECKING` imports of codec types (in `v3/_definition.py`,
`v3/data_type/string.py` and `v3/data_type/_byte.py`) become run-time
imports from `zarr_metadata.v3._types.codec`, and the codec defaults
spell their names with `BYTES_CODEC_NAME`, `VLEN_UTF8_CODEC_NAME` and
`VLEN_BYTES_CODEC_NAME`. The conflicts in the data type modules were
their import blocks: `EmptyConfiguration` comes from `_common`, and
`ConstantFillValue` from `definition._definition`.

Assisted-by: ClaudeCode:claude-opus-5-5
…lass

A codec was a `CodecDefinition(kind=...)`, and a table refused the
members a codec of a kind is never asked: the chunk rules, transition and
pipelines of a bytes -> bytes codec, and the transition of an array ->
bytes codec. Each kind is now a class: `ArrayArrayCodecDefinition`,
`ArrayBytesCodecDefinition` and `BytesBytesCodecDefinition`.
`ArrayCodecDefinition`, the base of the first two, holds `chunk_rules`
and `pipelines`, and only an array -> array codec has `transition`, so a
bytes -> bytes codec has no member that would be about a chunk.
`CodecDefinition` is their abstract base, whose `kind` property each
class gives, and it stays the kind that a scope and `resolve` name.

The pipeline and the strategies dispatch on the class instead of
comparing kind strings, so the type checker sees which codecs have which
members.

Assisted-by: ClaudeCode:claude-opus-5-5
`cast_value` and `scale_offset` decided which data types they accept
from lists of data type names: 25 names across three sets, 11 of them
float types this package does not define, kept only so `wrap` could
refuse them. A data type defined elsewhere was judged by whether its
name happened to be listed.

A `DataTypeDefinition` now declares what kind of numbers its values are
with `number_class`, a `NumberClass`: integral, floating-point, complex
or non-numeric. The two codecs' rules read it, as the `bytes` codec
reads `storage`: `cast_value` refuses complex and non-numeric data types
on either side of the cast and wraps only to one that is not
floating-point, and `scale_offset` refuses non-numeric ones. This is how
their specs state it ("data types that model real numbers", "an integral
type"), and neither a codec nor a data type has to name the other, so an
extension codec can read the class of a core data type and an extension
data type is judged by what it declares.

A data type that declares no number class is accepted, as an unknown
`storage` is, so nothing that was accepted is refused. Each of the 24
data types this package defines declares its class and is accepted or
refused as before, with the same messages.

Assisted-by: ClaudeCode:claude-fable-5-2
…cs do not list

The `cast_value` and `scale_offset` specs each name the integer, unsigned
integer and floating-point types as "the following data types defined in
this repository" that the codec supports. That repository also defines
the numpy time types and the complex types, and neither list names them.
The package accepted the time types for both codecs, on the strength of
the time types' own spec ("compatible with any codec that supports
arrays of signed 64-bit integers"), and accepted the complex types for
`scale_offset`, which its spec was read as leaving open.

The codec's spec says what the codec is defined for, so both codecs now
refuse `numpy.datetime64`, `numpy.timedelta64` and the complex types.
The time types declare the non-numeric number class, and `scale_offset`
refuses the same classes as `cast_value`, so one set serves both. zarrs
gives the same verdict for `cast_value`.

The strategies no longer draw these combinations, and the tests that
reached them are removed or moved to the table of refused data types.

Assisted-by: ClaudeCode:claude-fable-5-2
… after fork PR 445

The fragments take the number of the fork PR for now, and are renumbered
to the upstream PR that merges them.

Assisted-by: ClaudeCode:claude-fable-5-2
Upstream main gained zarr-developers#4627 as squash commit 901290d, whose tree is the
same as this branch's base (eb3061e). The nine conflicts were codec
modules that this branch changed after that base, so each keeps this
branch's version, and the merge leaves the tree unchanged.

Assisted-by: ClaudeCode:claude-opus-5-5
@github-actions github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package labels Nov 12, 2026
@read-the-docs-community

read-the-docs-community Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.90%. Comparing base (e8e8d96) to head (6e7bcfe).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4628   +/-   ##
=======================================
  Coverage   94.90%   94.90%           
=======================================
  Files          96       96           
  Lines       14051    14051           
=======================================
  Hits        13335    13335           
  Misses        716      716           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…not plain numbers

The open zarr-extensions proposals include an optional (nullable) data
type, a fixed-size list and a ragged ndarray. Their values hold numbers
without being numbers, and neither arithmetic codec's spec lists them, so
they belong in `non_numeric`, which said "not numbers at all". It now
says every value that is not a plain number, naming records, lists and
values that may be missing.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
d-v-b marked this pull request as ready for review November 12, 2026 22:06
@d-v-b
d-v-b merged commit 4dc5b01 into zarr-developers:main Nov 12, 2026
45 checks passed
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 12, 2026
Shortens the zarr-developers#4621 and zarr-developers#4628 feature entries and the zarr-developers#4628 bugfix entry
from 254, 216 and 111 words to 132, 140 and 43. The dropped detail, such
as the float 0.0 spelling, how create_default reports a member given
with a problem, the bfloat16 example and the argument from the spec text
for refusing the time and complex types, stays in those pull requests.

Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b added a commit that referenced this pull request Nov 13, 2026
* chore(zarr-metadata): build 0.8.0 changelog

Assisted-by: ClaudeCode:claude-opus-5-5

* docs(zarr-metadata): condense the long 0.8.0 release notes

Shortens the #4621 and #4628 feature entries and the #4628 bugfix entry
from 254, 216 and 111 words to 132, 140 and 43. The dropped detail, such
as the float 0.0 spelling, how create_default reports a member given
with a problem, the bfloat16 example and the argument from the spec text
for refusing the time and complex types, stays in those pull requests.

Assisted-by: ClaudeCode:claude-opus-5-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant