Repository navigation
Conversation
Every extension module held its types (name constants, `Literal` names, configuration and object `TypedDict`s, metadata aliases) beside its definition. Codec modules imported data type modules at run time, so a data type that needed a codec's types could import them only under `TYPE_CHECKING`. The types now live in modules that hold types only and import no definition: one module per kind under `v3/_types/` (codec, data_type, chunk_grid, chunk_key_encoding, consolidated) and `v2/_types/` (codec, data_type). Their packages' `__init__`s hold no definitions, so importing a types module never runs one. `EmptyConfiguration` moves to `_common`, and `ZarrV2DataTypeDeclaration` to `v2/_declaration_json`. Rules and other functions stay beside their definition, which imports its types from the types module. Each public extension module re-exports its types, so every public import path is unchanged. The codec rules that name data types (`_arithmetic`, `vlen_utf8`, `vlen_bytes`, `sharding_indexed`) import those names from the types module. Assisted-by: ClaudeCode:claude-opus-5-5
Each types-only module is imported in a fresh interpreter in which the `zarr_metadata`, `zarr_metadata.v2` and `zarr_metadata.v3` packages (whose `__init__`s import every definition) are empty stand-ins. The test asserts that no module holding a `Definition` ran. Every other package `__init__` runs for real, so a types module placed in a package that exports definitions fails. A control test checks that the probe does see a definition module when an extension module is imported. Assisted-by: ClaudeCode:claude-opus-5-5
…r-developers#4627 Assisted-by: ClaudeCode:claude-opus-5-5
Brings in zarr-developers#4621, which added `array_bytes_codec_default` and `fill_value_default` to the data type definitions. Its three `TYPE_CHECKING` imports of codec types (in `v3/_definition.py`, `v3/data_type/string.py` and `v3/data_type/_byte.py`) become run-time imports from `zarr_metadata.v3._types.codec`, and the codec defaults spell their names with `BYTES_CODEC_NAME`, `VLEN_UTF8_CODEC_NAME` and `VLEN_BYTES_CODEC_NAME`. The conflicts in the data type modules were their import blocks: `EmptyConfiguration` comes from `_common`, and `ConstantFillValue` from `definition._definition`. Assisted-by: ClaudeCode:claude-opus-5-5
…lass A codec was a `CodecDefinition(kind=...)`, and a table refused the members a codec of a kind is never asked: the chunk rules, transition and pipelines of a bytes -> bytes codec, and the transition of an array -> bytes codec. Each kind is now a class: `ArrayArrayCodecDefinition`, `ArrayBytesCodecDefinition` and `BytesBytesCodecDefinition`. `ArrayCodecDefinition`, the base of the first two, holds `chunk_rules` and `pipelines`, and only an array -> array codec has `transition`, so a bytes -> bytes codec has no member that would be about a chunk. `CodecDefinition` is their abstract base, whose `kind` property each class gives, and it stays the kind that a scope and `resolve` name. The pipeline and the strategies dispatch on the class instead of comparing kind strings, so the type checker sees which codecs have which members. Assisted-by: ClaudeCode:claude-opus-5-5
`cast_value` and `scale_offset` decided which data types they accept
from lists of data type names: 25 names across three sets, 11 of them
float types this package does not define, kept only so `wrap` could
refuse them. A data type defined elsewhere was judged by whether its
name happened to be listed.
A `DataTypeDefinition` now declares what kind of numbers its values are
with `number_class`, a `NumberClass`: integral, floating-point, complex
or non-numeric. The two codecs' rules read it, as the `bytes` codec
reads `storage`: `cast_value` refuses complex and non-numeric data types
on either side of the cast and wraps only to one that is not
floating-point, and `scale_offset` refuses non-numeric ones. This is how
their specs state it ("data types that model real numbers", "an integral
type"), and neither a codec nor a data type has to name the other, so an
extension codec can read the class of a core data type and an extension
data type is judged by what it declares.
A data type that declares no number class is accepted, as an unknown
`storage` is, so nothing that was accepted is refused. Each of the 24
data types this package defines declares its class and is accepted or
refused as before, with the same messages.
Assisted-by: ClaudeCode:claude-fable-5-2
…cs do not list
The `cast_value` and `scale_offset` specs each name the integer, unsigned
integer and floating-point types as "the following data types defined in
this repository" that the codec supports. That repository also defines
the numpy time types and the complex types, and neither list names them.
The package accepted the time types for both codecs, on the strength of
the time types' own spec ("compatible with any codec that supports
arrays of signed 64-bit integers"), and accepted the complex types for
`scale_offset`, which its spec was read as leaving open.
The codec's spec says what the codec is defined for, so both codecs now
refuse `numpy.datetime64`, `numpy.timedelta64` and the complex types.
The time types declare the non-numeric number class, and `scale_offset`
refuses the same classes as `cast_value`, so one set serves both. zarrs
gives the same verdict for `cast_value`.
The strategies no longer draw these combinations, and the tests that
reached them are removed or moved to the table of refused data types.
Assisted-by: ClaudeCode:claude-fable-5-2
… after fork PR 445 The fragments take the number of the fork PR for now, and are renumbered to the upstream PR that merges them. Assisted-by: ClaudeCode:claude-fable-5-2
Upstream main gained zarr-developers#4627 as squash commit 901290d, whose tree is the same as this branch's base (eb3061e). The nine conflicts were codec modules that this branch changed after that base, so each keeps this branch's version, and the merge leaves the tree unchanged. Assisted-by: ClaudeCode:claude-opus-5-5
…elopers#4626) * chore(deps-dev): bump the python-dependencies group across 1 directory with 13 updates Bumps the python-dependencies group with 13 updates in the / directory: | Package | From | To | | --- | --- | --- | | [obstore](https://github.com/geospatial-jeff/pyasyncio-benchmark) | `0.12.0` | `0.12.1` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.16.2` | `7.17.2` | | [pytest](https://github.com/pytest-dev/pytest) | `9.1.1` | `9.2.0` | | [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) | `1.4.0` | `1.4.1.post1` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.170.5` | `6.170.14` | | [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) | `5.3.0` | `5.3.1` | | [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed) | `5.0.3` | `5.0.4` | | [uv](https://github.com/astral-sh/uv) | `0.13.5` | `0.13.7` | | [moto[s3,server]](https://github.com/getmoto/moto) | `5.2.4` | `5.2.5` | | [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.7` | `9.8.0` | | [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.5` | `1.12.6` | | [ruff](https://github.com/astral-sh/ruff) | `0.17.1` | `0.17.3` | | [mypy](https://github.com/python/mypy) | `2.4.0` | `2.4.1` | Updates `obstore` from 0.12.0 to 0.12.1 - [Commits](https://github.com/geospatial-jeff/pyasyncio-benchmark/commits) Updates `coverage` from 7.16.2 to 7.17.2 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.16.2...7.17.2) Updates `pytest` from 9.1.1 to 9.2.0 - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@9.1.1...9.2.0) Updates `pytest-asyncio` from 1.4.0 to 1.4.1.post1 - [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases) - [Commits](pytest-dev/pytest-asyncio@v1.4.0...v1.4.1.post1) Updates `hypothesis` from 6.170.5 to 6.170.14 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@v6.170.5...v6.170.14) Updates `pytest-benchmark` from 5.3.0 to 5.3.1 - [Release notes](https://github.com/ionelmc/pytest-benchmark/releases) - [Changelog](https://github.com/ionelmc/pytest-benchmark/blob/master/CHANGELOG.rst) - [Commits](ionelmc/pytest-benchmark@v5.3.0...v5.3.1) Updates `pytest-codspeed` from 5.0.3 to 5.0.4 - [Release notes](https://github.com/CodSpeedHQ/pytest-codspeed/releases) - [Changelog](https://github.com/CodSpeedHQ/pytest-codspeed/blob/master/CHANGELOG.md) - [Commits](CodSpeedHQ/pytest-codspeed@v5.0.3...v5.0.4) Updates `uv` from 0.13.5 to 0.13.7 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](astral-sh/uv@0.13.5...0.13.7) Updates `moto[s3,server]` from 5.2.4 to 5.2.5 - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@5.2.4...5.2.5) Updates `mkdocs-material[imaging]` from 9.7.7 to 9.8.0 - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](squidfunk/mkdocs-material@9.7.7...9.8.0) Updates `markdown-exec[ansi]` from 1.12.5 to 1.12.6 - [Release notes](https://github.com/pawamoy/markdown-exec/releases) - [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md) - [Commits](pawamoy/markdown-exec@1.12.5...1.12.6) Updates `ruff` from 0.17.1 to 0.17.3 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.17.1...0.17.3) Updates `mypy` from 2.4.0 to 2.4.1 - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v2.4.0...v2.4.1) --- updated-dependencies: - dependency-name: obstore dependency-version: 0.12.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: coverage dependency-version: 7.17.2 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: pytest dependency-version: 9.2.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: pytest-asyncio dependency-version: 1.4.1.post1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: hypothesis dependency-version: 6.170.14 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: pytest-benchmark dependency-version: 5.3.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: pytest-codspeed dependency-version: 5.0.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.13.7 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: moto[s3,server] dependency-version: 5.2.5 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mkdocs-material[imaging] dependency-version: 9.8.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: markdown-exec[ansi] dependency-version: 1.12.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: ruff dependency-version: 0.17.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mypy dependency-version: 2.4.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * test: keep the suite running with pytest 9.2 and hypothesis 6.170.7+ pytest 9.2 applies the configured filterwarnings while importing plugins (pytest#12697), so the "error" filter turns the ast.Num DeprecationWarning that astor (a dependency of pytest-accept) triggers at import on Python < 3.14 into a startup failure (exit code 3). Ignore that one astor-scoped deprecation, which pytest <= 9.1 never routed through the filters. pytest 9.2 also requires packaging>=24, which conflicts with the min_deps pin packaging==22.* (the declared runtime floor packaging>=22.0). Keep that floor tested by overriding pytest to <9.2 in the min_deps env only, via UV_OVERRIDE and ci/min_deps-overrides.txt; every other env uses pytest 9.2.0. hypothesis 6.170.7 annotates run_state_machine_as_test as taking type[RuleBasedStateMachine] (its docstring still accepts any zero-arg factory), so mypy now reports arg-type instead of no-untyped-call at the four call sites that pass a factory function. Update the ignore codes; no test logic changes. * test: drop the unused pytest-accept plugin pytest-accept was added in zarr-developers#2589 for a `hatch run doctest:fix` script that ran `pytest --accept` over the RST doctests. That script and those doctests are gone, and nothing in the justfile or CI passes `--accept`. The plugin's only remaining effect was importing astor (last released in 2019) at every pytest startup, whose `ast.Num` lookup pytest 9.2 turns into a startup error under our "error" warning filter. Removing the plugin removes astor, so the astor-scoped filter added for pytest 9.2 is no longer needed. Assisted-by: ClaudeCode:claude-opus-5-5 * build(deps): require packaging>=24.0 pytest 9.2 requires packaging>=24, so the min_deps env could no longer pin packaging==22.*. Instead of holding min_deps on pytest<9.2 through a UV_OVERRIDE file, raise the runtime floor to packaging>=24.0 (released March 2024) and test that floor in min_deps. zarr uses only packaging.version.Version and parse, which behave the same on 22 and 24. min_deps now runs the same pytest as every other env, and the override file and env var are gone. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: add a changelog entry for the packaging>=24.0 floor Assisted-by: ClaudeCode:claude-opus-5-5 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: invisiblefix-bot <355452247+invisiblefix-bot@users.noreply.github.com>
… after zarr-developers#4628 Assisted-by: ClaudeCode:claude-opus-5-5
* fix: migrate v2 arrays whose fill_value is null Zarr v2 allows "fill_value": null, and xarray writes it for variables without a _FillValue. Zarr v3 requires a fill value, so `zarr migrate v3` raised a TypeError on such arrays. Use the data type's default scalar, which is what zarr-python already returns for unwritten chunks of the v2 array. * docs: add changelog entry for zarr-developers#4616 * test(migrate): cover a partly written chunk in the null fill value test Write only part of the first chunk, so the test checks that the padding stored in a partial chunk reads back the same after migration, and compare the v2 read against the same expected values as the v3 read. Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Davis Vann Bennett <davis.v.bennett@gmail.com>
…not plain numbers The open zarr-extensions proposals include an optional (nullable) data type, a fixed-size list and a ragged ndarray. Their values hold numbers without being numbers, and neither arithmetic codec's spec lists them, so they belong in `non_numeric`, which said "not numbers at all". It now says every value that is not a plain number, naming records, lists and values that may be missing. Assisted-by: ClaudeCode:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Breaking: a v3 codec is defined with the class of its kind,
ArrayArrayCodecDefinition,ArrayBytesCodecDefinitionorBytesBytesCodecDefinition, instead ofCodecDefinition(kind=...). The kind is now the class'skindproperty, andCodecDefinitionis their abstract base, which a scope andresolvestill use to name the kind. A codec that receives an array, anArrayCodecDefinition, haschunk_rulesandpipelines, an array -> array codec also hastransition, and a bytes -> bytes codec, which receives bytes, has none of them, so building one with any of them is aTypeErrorfrom the class itself.A v3 data type definition declares what kind of numbers its values are with
number_class, aNumberClass:"integral","floating_point","complex"or"non_numeric".cast_valueandscale_offsetread it to decide which data types they accept, where each kept a list of data type names, so a data type defined elsewhere is judged by what it declares: both codecs refuse one that declares complex or non-numeric values, andcast_valuerefuses towrapto one that declares floating-point values. A data type that declares no number class is accepted by both, as before. A data type defined elsewhere under one of the float names thecast_valuespec lists, such asbfloat16, was refused as awraptarget by its name, and is now refused once it declares"floating_point".Breaking:
cast_valueandscale_offsetrefuse the data types their specs do not list. Each spec names the integer and floating-point types as "the following data types defined in this repository" that it supports, and that repository also defines the numpy time types and the complex types. So both codecs now refusenumpy.datetime64andnumpy.timedelta64, andcast_valuerefuses them as the data type it casts to, andscale_offsetnow refusescomplex64andcomplex128. All of these were accepted. The time types' own spec says they are "compatible with any codec that supports arrays of signed 64-bit integers", which the two codecs' specs do not bear out.zarr_metadata.testing.strategiesno longer draws these combinations.Agent summary
Three commits, each of which passes
just checkon its own.1. One class per codec kind (
3348a561d)CodecDefinitionwas built withkind="array_array" | "array_bytes" | "bytes_bytes", and a table (_UNASKED) refused the members a codec of a kind is never asked: the chunk rules, transition and pipelines of a bytes -> bytes codec, and the transition of an array -> bytes codec.ArrayCodecDefinitionholdschunk_rulesandpipelines,ArrayArrayCodecDefinitionaddstransition, andBytesBytesCodecDefinitionhas none of them.CodecDefinitionis an abstract base (ABC, withkindan abstract property), and it is still the kind that scopes andresolvename.BytesBytesCodecDefinition(name='gzip'), so the error messages that include it changed.2.
number_classon data type definitions (8d8e730ec)cast_valueandscale_offsetdecided which data types they accept from three sets holding 25 data type names, 11 of them float types this package does not define, kept so thatwrapcould refuse them.storage, and the two codecs' own rules read it. Their specs state the criterion the same way ("data types that model real numbers", "an integral type").None, the default, means unknown, and both codecs accept it, as thebytescodec accepts an unknownstorage.3. Verdicts that follow the codec specs (
28e0bfd0a)"non_numeric", andscale_offsetrefuses the same classes ascast_value, so one set of refused classes serves both codecs.scale_offset's message is now "expected a chunk of an integer or floating-point data type, got ...", since "a data type with arithmetic" would be wrong for complex numbers.Design notes
supported_codecsset of codec names and codecs arequires_data_type_supportflag, modeled on zarrs'sdefine_data_type_support!. In zarrs the per-codec trait carries behavior and properties (cast_value_is_integral()), and either crate can add a pair. As a name list it lost both:cast_valuestill needed a list of float names forwrap, and an extension codec could not accept coreint8without redefining it. The agent replaced it after a review of the design.scale_offsetspec's list covers only the data types of its own repository, and it leaves out all 13 complex types defined there. The agent read that as "no complex numbers", so a data type defined elsewhere that declares"complex"is refused byscale_offsettoo. The alternative is to refuse onlycomplex64andcomplex128by name.cast_value, so it gives the same verdict for the time types. It has noscale_offset. The agent read the zarrs source atdb0f278and did not run it, and did not check tensorstore.Testing
tests/v3/test_number_class.py: a table pins the number class of each of the 24 data types this package defines. A second table runs an extension data type of each class, and one that declares none, through both codecs, as input and ascast_value's target with and withoutwrap, and checks where each problem is reported and its message. One error test covers anumber_classthat is not one of the values.tests/v3/test_pipelines.py: complex and both time types joined the table of data types each arithmetic codec refuses. The two cases that pinned the old acceptance are removed.tests/testing/test_strategies.py: the test that reached complexscale_offsetvalues is removed, and the tests of complex scales and time-type cast targets lost those cases.just checkpasses on the head commit: ruff, pyright (0 errors), the suite on Python 3.11 and on the newest supported version, andmkdocs build --strict.Stack and bookkeeping
zarr-developers/zarr-python#4627(squash commit901290de8). The fork'smainwas synced to upstream, and upstreammainwas merged into this branch (22e65d51b). The squash has the same tree as this branch's old base, so the merge kept this branch's side of the nine conflicting codec modules and changed no file.53d57d210,bcc20c872,c12e4e405,eb3061e05), sincemainholds them as one squash; they add nothing to the diff. The commits that belong to this PR are3348a561d,8d8e730ec,28e0bfd0aand199ceb595.4628.feature.md,4628.bugfix.md).🤖 Generated with Claude Code