feat(zarr-metadata)!: read metadata against definitions in a scope, and make every model a pair of document and scope

.github/workflows/zarr-metadata.yml

@@ -9,16 +9,20 @@ on:
     paths:
       - 'packages/zarr-metadata/**'
       - '.github/workflows/zarr-metadata.yml'
-      # The docs config and header script shared by all package sites.
+      # The docs config and header script shared by all package sites, and
+      # zarr-python's docs hook, which this site uses.
       - 'packages/mkdocs-base.yml'
       - 'packages/source-version.js'
+      - 'mkdocs_hooks.py'
   pull_request:
     paths:
       - 'packages/zarr-metadata/**'
       - '.github/workflows/zarr-metadata.yml'
-      # The docs config and header script shared by all package sites.
+      # The docs config and header script shared by all package sites, and
+      # zarr-python's docs hook, which this site uses.
       - 'packages/mkdocs-base.yml'
       - 'packages/source-version.js'
+      - 'mkdocs_hooks.py'
   workflow_dispatch:
 
 permissions:
@@ -102,7 +106,7 @@ jobs:
           repo: casey/just
           version: 1.58.0
       - name: Run pyright
-        # The pyright version and interpreter pins live in the justfile.
+        # Pyright runs unpinned; the interpreter pin lives in the justfile.
         run: just typecheck
 
   docs:

packages/zarr-metadata/changes/4365.bugfix.md

@@ -1 +1 @@
-The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are tolerated and dropped (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The Pydantic schema follows: `ZarrV2ArrayMetadataJSON` is open and the filter pipeline has no minimum length.
+The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are accepted when they are JSON values with string keys, and are kept and written back by `to_json` and `to_key_value`; any other member is a problem, as in a v3 document (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The JSON Schema that `zarr_metadata.pydantic` generates follows: it accepts other members, and the filter pipeline has no minimum length.

packages/zarr-metadata/changes/4378.bugfix.md

@@ -20,6 +20,5 @@ type checker rejected putting them into the very fields they document
 declared shape no longer satisfy the closed types, and `name` /
 `configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`.
 Both were previously accepted by type checkers but produced documents
-outside the spec's shapes. `zarr_metadata.pydantic` serializers now
-declare their return schema via the pydantic-facing shadow types, so
-pydantic schema generation stays warning-free.
+outside the spec's shapes. Generating a pydantic JSON Schema for
+the `zarr_metadata.pydantic` field types no longer emits warnings.

packages/zarr-metadata/changes/4378.misc.md

@@ -1,6 +1,4 @@
-**Breaking:** every `validate_*` function in `zarr_metadata.model` now
-returns `tuple[ValidationProblem, ...]` instead of `list[ValidationProblem]`,
-and `MetadataValidationError.problems` is a tuple. Iteration and indexing are
-unchanged; callers that mutate reports must first copy them with
-`list(problems)`. `MetadataValidationError` still accepts any problem
-sequence.
+**Breaking:** `MetadataValidationError.problems` is now a tuple instead of
+a list. Iteration and indexing are unchanged; code that changes a report
+must first copy it with `list(problems)`. `MetadataValidationError` still
+accepts any problem sequence.

packages/zarr-metadata/changes/4420.bugfix.md

@@ -6,18 +6,14 @@ the model refused such a document and could not read the store at all.
 A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2
 `.zmetadata`, and the attributes of the nodes a v3 group's inline
 `consolidated_metadata` holds may now hold `NaN`, `Infinity` and
-`-Infinity`, in every document `validate_*`, `parse_*` and `is_*`, and in
-`from_key_value` and `to_key_value`. Wherever the spec interprets a value
-it spells those numbers as strings, so anywhere else a non-finite number
-is still refused, and the store reader now says where one is instead of
-failing to decode the document. `is_json`, `validate_json` and
-`parse_json` judge by RFC 8259 alone, so a document the node guards
-accept can hold a value `is_json` refuses.
+`-Infinity`, in every reader, such as `read_array_metadata_v3`, and in
+each model's `from_json`, `from_key_value` and `to_key_value`. Wherever
+the spec interprets a value it spells those numbers as strings, so
+anywhere else a non-finite number is still refused, and the store reader
+now says where one is instead of failing to decode the document.
 
-`to_key_value` validates the document it writes as `from_key_value`
-validates what it reads, and raises `MetadataValidationError` with every
-problem instead of writing a document the reader refuses. A model built
-by hand is not validated, so the writer is where these are caught:
+A model whose document the reader refuses raises
+`MetadataValidationError` with every problem, instead of being written:
 
 - a non-finite number outside attributes, which was refused before too;
 - a non-string attribute key, which was written as a string;

packages/zarr-metadata/changes/4423.bugfix.md

@@ -1,7 +1,7 @@
 A `bytes` or `bytearray` is not a JSON array. Where a document expects an
-array, an empty one passed as an array with no items:
-`parse_array_metadata_v3` returned `storage_transformers=b""`, and the v2
-readers took `dtype=b""` and `filters=b""`, then failed when the model was
-written. The validators and `parse_*` now refuse one at `codecs`,
-`storage_transformers`, `dimension_names`, a v2 `dtype` and its field
-records, and v2 `filters`.
+array, an empty one passed as an array with no items: the v3 parser
+returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`,
+then failed when the model was written. The readers and models now refuse
+one at `storage_transformers` and at a v2 `dtype` and its field records. At
+`codecs`, `dimension_names` and v2 `filters`, which already refused one,
+the problem now says the value has the wrong type.

packages/zarr-metadata/changes/4430.misc.md

@@ -1,4 +1,4 @@
-`ZarrV2ZGroupJSON`, `ZarrV2GroupMetadataJSON` and its Partial are closed
+`ZarrV2ZGroupJSON` and `ZarrV2GroupMetadataJSON` are closed
 (PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group
 validator already said, and as its pydantic schema already was. A type
 checker now refuses a v2 group document with a key it does not declare.

packages/zarr-metadata/changes/4432.feature.md

@@ -11,4 +11,4 @@ says neither; and each annotation is evaluated in the module of the class
 that wrote it. A key a closed TypedDict does not declare is reported, as
 the new problem kind `unknown_key`, and left out of the value, which still
 comes back; a union of TypedDicts tagged by a `Literal` key is read by the
-branch the tag names. `typeddict_keys` says what a TypedDict declares.
+branch the tag names.

packages/zarr-metadata/changes/4434.feature.1.md

@@ -5,7 +5,6 @@ gives wrong bytes as surely as ignoring a data type gives wrong values,
 and the spec naming only three points is read as an oversight. A document
 that declares a codec or a storage transformer ignorable now has a problem
 at `codecs.N.must_understand` or `storage_transformers.N.must_understand`,
-so the readers refuse it, and `to_key_value` refuses to write a model that
-holds one. The JSON schema `zarr_metadata.pydantic` generates for an array
-document says the same. A metadata field read on its own still takes
-`must_understand: false`.
+so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array
+document says the same, and so does `resolve`, which reads one field on
+its own.

packages/zarr-metadata/changes/4434.feature.md

@@ -1,35 +1,34 @@
 Each extension's metadata is a definition: a value holding the name the
 metadata carries, the TypedDict its configuration is -- the one
 declaration of that JSON -- and plain functions over that TypedDict for
-the rules. `zarr_metadata.v3.definition` is the door, and its module
-docstring the guide. Every extension the package models is one: the
+the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the
+doors, and their module docstrings the guide. Every extension the package models is one: the
 codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`,
 `sharding_indexed`, `transpose` and `zstd`; every data type, raw bits
-included (`r16` reads as `r*` with `{"bits": 16}`, the size its name
-carries); the `regular` and `rectilinear` chunk grids; and the `default`
+included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size
+its name carries); the `regular` and `rectilinear` chunk grids; and the `default`
 and `v2` chunk key encodings. Each is a constant in its own module --
 `GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the
 specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions`
 registers besides. A third party writes a TypedDict, a rule function and
-a definition, and extends a scope with it; a definition checks itself
+a definition, and adds it to a scope; a definition checks itself
 when built, and nothing happens at class creation.
 
 Reading a field is three steps, each usable on its own by a caller
 holding nothing but JSON. `check(value, SomeTypedDict)`, from
 `zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs
-nothing else; `definition.judge(configuration)` is the check, each nested
+nothing else; `definition.read_configuration(configuration)` is the check, each nested
 field's envelope judged, and then the rules; `resolve(field,
 CodecDefinition, scope)` reads a whole field in a scope -- its envelope
 judged, its name related to a definition, its configuration judged, each
 nested field read the same way -- and returns what the scope made of it,
-`Resolved`, with every problem located. A name nothing in scope claims is
-`out_of_scope` and left unjudged. A shard's pipelines, a cast's target
-type and a struct's field types are nested fields, annotated `CodecField`
-and `DataTypeField`, and read in the scope their field is read in.
-`configuration_of(resolved, GZIP_CODEC)` is a read configuration typed as
-its definition's TypedDict. A codec definition also says whether the size
+with every problem located. A name nothing in scope claims is left
+unjudged. A shard's pipelines, a cast's target
+type and a struct's field types are nested fields, annotated `ZarrV3CodecField`
+and `ZarrV3DataTypeField`, and read in the scope their field is read in.
+A codec definition also says whether the size
 of what it gives out is `static` or `dynamic`, and a member annotated
-`StaticCodecField` takes codecs of static size only: a shard's
+`ZarrV3StaticCodecField` takes codecs of static size only: a shard's
 `index_codecs`, where the spec forbids a compressor, so a `gzip` there is a
 problem at its place.
 
@@ -41,11 +40,11 @@ what the rules are handed. A definition refuses a TypedDict that says
 nothing of the keys it does not declare, and a member typed
 `ZarrV3MetadataFieldJSON`, which would check as plain JSON.
 
-`canonicalize(field, kind, scope)` gives a field without problems in its
+`canonical_of(field, problems)` gives a field without problems in its
 simplest equivalent spelling: each nested field in its own simplest
 spelling, then its definition's own `canonical`, and the envelope in the
-fewest words -- the bare name when nothing is configured, and no
-`must_understand`. A field with any problem, an unknown key included, has
+fewest words every reader takes, with no `must_understand`. A field with
+any problem, an unknown key included, has
 none, since a simpler spelling would erase what its author wrote. What
 `canonical` gives is judged again, and one that does not hold is a
 `ValueError`, a fault in the definition.

packages/zarr-metadata/changes/4436.feature.md

@@ -1,13 +1,12 @@
-**Breaking:** the v3 validators read each extension point through its
-definition: `validate_array_metadata_v3` judges the data type, the chunk
+**Breaking:** the v3 readers read each extension point through its
+definition: `read_array_metadata_v3` judges the data type, the chunk
 grid, the chunk key encoding, each codec and each storage transformer
 against the definition that claims its name in a scope,
 `CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of
 99, or a key a codec does not declare, is a problem located in the
 configuration, and a document holding one, accepted before, is refused.
-So do the `is_*` and `parse_*` beside it and the v3 group validators,
-with each array an inline `consolidated_metadata` holds, and the v3
-model classes' `from_json`, `from_key_value` and `to_key_value` take the
-same `context`, so a model read in a scope is written in it. The
-pydantic types read in `CORE_AND_EXTENSIONS`. A name nothing in scope
-claims is left unjudged.
+So do the v3 group readers, and the v3 model classes' `from_json` and
+`from_key_value` take the same `context`.
+A name nothing in scope claims is left unjudged. A problem in a field
+that a configuration holds, such as a codec inside a shard, is reported
+at that field's own location, and the field that holds it is still read.

packages/zarr-metadata/changes/4443.feature.2.md

@@ -6,5 +6,5 @@ struct fill value missing a field are each a problem at `fill_value`,
 where the package accepted them before. `fill_value_problems(data_type,
 value)` judges a fill value against a data type field a scope read, and
 a field that is read keeps the fields it read inside as
-`Resolved.nested`, a `Nested` mapping by location. A data type nothing
+`nested`, a `Nested` mapping by location. A data type nothing
 in scope claims leaves its fill value unjudged.

packages/zarr-metadata/changes/4443.feature.3.md

@@ -1,7 +1,6 @@
 **Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by
 the grid's definition. A regular grid whose `chunk_shape` does not have
-one length per dimension of the shape, or has a length of 0 for a
-dimension that is not empty, and a rectilinear grid whose `chunk_shapes`
-does not have one entry per dimension, or whose chunk lengths fall short
-of their dimension, each have a problem in `chunk_grid.configuration`,
-where the package accepted them before.
+one length per dimension of the shape, and a rectilinear grid whose
+`chunk_shapes` does not have one entry per dimension, or whose chunk
+lengths fall short of their dimension, each have a problem in
+`chunk_grid.configuration`, where the package accepted them before.

packages/zarr-metadata/changes/4443.feature.4.md

@@ -11,10 +11,10 @@ not integral, or mapping a scalar that is not a fill value of the data
 type on its side; a `scale_offset` handed values that are no numbers, or
 with an `offset` or `scale` that is not a value of its data type; and a
 struct field whose values vary in size are each a problem where they
-sit, where the package accepted them before. `read_pipeline(codecs,
-chunk)` gives each codec with the chunk it is handed,
-`chunk_grid_lengths(grid, shape)` the lengths a grid's chunks take, and
-`storage_of(data_type)` how a data type's values are stored. A
+sit, where the package accepted them before. A reading records each
+codec with the chunk it is handed, as a `Stage`, and the lengths a
+grid's chunks take; `storage_of(data_type)` says how a data type's
+values are stored. A
 definition's `rules` are handed the fields its configuration holds as
 the scope read them, and the kinds gain `chunk_lengths` (grids),
 `storage` (data types), and `chunk_rules` and `transition` (codecs).

packages/zarr-metadata/changes/4490.feature.md

@@ -0,0 +1,17 @@
+**Breaking:** metadata is now read against the definitions in a scope, a
+`ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs,
+data types and other fields it supports, and a model is the document
+together with the scope it was read in. The `read_*` readers report every
+problem at its location, `unclaimed_fields` lists each field the scope
+does not know, so a reader can tell whether it can open a node, repair
+readers undo known zarr-python writer bugs, and JSON Schemas are exported.
+Code written for 0.5.0 must change: models are no longer dataclasses,
+their field members hold `AcceptedField` or `UnclaimedField` values,
+`update` takes JSON and takes members the spec does not define through
+`extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for
+documents and the `...Partial` TypedDicts are removed, some names moved to
+another module, the v2 constants and models follow the v3 names and
+shapes, and a v2 time dtype without a unit is refused. `to_json` now
+writes each field as it was read, rather than shortening a codec to a bare
+name that zarr-python could not open, and zarr-metadata now depends on
+`annotated-types`.

packages/zarr-metadata/docs/api/definition.md

@@ -0,0 +1,8 @@
+---
+title: definition
+---
+
+::: zarr_metadata.definition
+    options:
+      inherited_members: false
+      show_if_no_docstring: false