Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions packages/zarr-metadata/changes/4628.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
**Breaking:** `cast_value` and `scale_offset` refuse the data types their
specs do not list. Each spec names the integer and floating-point types as
"the following data types defined in this repository" that it supports, and
that repository also defines the numpy time types and the complex types. So
both codecs now refuse `numpy.datetime64` and `numpy.timedelta64`, and
`cast_value` refuses them as the data type it casts to, and `scale_offset`
now refuses `complex64` and `complex128`. All of these were accepted. The time
types' own spec says they are "compatible with any codec that supports arrays
of signed 64-bit integers", which the two codecs' specs do not bear out.
`zarr_metadata.testing.strategies` no longer draws these combinations.
21 changes: 21 additions & 0 deletions packages/zarr-metadata/changes/4628.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
**Breaking:** a v3 codec is defined with the class of its kind,
`ArrayArrayCodecDefinition`, `ArrayBytesCodecDefinition` or
`BytesBytesCodecDefinition`, instead of `CodecDefinition(kind=...)`. The kind
is now the class's `kind` property, and `CodecDefinition` is their abstract
base, which a scope and `resolve` still use to name the kind. A codec that
receives an array, an `ArrayCodecDefinition`, has `chunk_rules` and
`pipelines`, an array -> array codec also has `transition`, and a bytes ->
bytes codec, which receives bytes, has none of them, so building one with any
of them is a `TypeError` from the class itself.

A v3 data type definition declares what kind of numbers its values are with
`number_class`, a `NumberClass`: `"integral"`, `"floating_point"`,
`"complex"` or `"non_numeric"`. `cast_value` and `scale_offset` read it to
decide which data types they accept, where each kept a list of data type
names, so a data type defined elsewhere is judged by what it declares: both
codecs refuse one that declares complex or non-numeric values, and
`cast_value` refuses to `wrap` to one that declares floating-point values. A
data type that declares no number class is accepted by both, as before. A data
type defined elsewhere under one of the float names the `cast_value` spec
lists, such as `bfloat16`, was refused as a `wrap` target by its name, and is
now refused once it declares `"floating_point"`.
22 changes: 14 additions & 8 deletions packages/zarr-metadata/docs/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ from typing_extensions import TypedDict

from zarr_metadata import ValidationProblem
from zarr_metadata.definition import Nested
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, BytesBytesCodecDefinition, ZarrV3Context


class AcmeLz4Configuration(TypedDict, closed=True):
Expand All @@ -51,10 +51,9 @@ def acme_lz4_rules(
)


ACME_LZ4 = CodecDefinition(
ACME_LZ4 = BytesBytesCodecDefinition(
name="acme.lz4",
configuration=AcmeLz4Configuration,
kind="bytes_bytes",
size="dynamic",
rules=acme_lz4_rules,
)
Expand Down Expand Up @@ -110,13 +109,20 @@ Other kinds declare more:
`create_default` is given no codecs (`array_bytes_codec_default`, such
as `vlen-utf8` for `string`). A data type whose values vary in size
declares its own, since the default, `bytes`, encodes values of a fixed
size.
size. It declares what kind of numbers its values are (`number_class`:
`"integral"`, `"floating_point"`, `"complex"` or `"non_numeric"`), which
the codecs that are defined only for some kinds of numbers read:
`cast_value` and `scale_offset` refuse a data type that declares
complex or non-numeric values. A data type that does not declare it is
accepted by both.
- A chunk grid declares which shapes it fits (`shape_rules`) and its chunk
lengths (`chunk_lengths`).
- A codec declares:
- what it checks about the chunk it receives (`chunk_rules`),
- what it hands on (`transition`),
- any pipelines it contains (`pipelines`).
- A codec is defined by the class of its kind, `ArrayArrayCodecDefinition`,
`ArrayBytesCodecDefinition` or `BytesBytesCodecDefinition`. A codec that
receives an array declares what it checks about the chunk it receives
(`chunk_rules`) and any pipelines it contains (`pipelines`), and an
array -> array codec also declares what it hands on (`transition`). A
bytes -> bytes codec receives bytes, so it declares none of these.

The [`zarr_metadata.v3.definition`](api/v3/definition.md) docstring
describes each of these.
6 changes: 3 additions & 3 deletions packages/zarr-metadata/docs/parsing-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,12 +258,12 @@ To make an unknown data type or codec known, add its definition to the context's

```python test="true" session="parsing-validation"
from zarr_metadata.definition import EmptyConfiguration
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, BytesBytesCodecDefinition, ZarrV3Context

# The definition of the codec that was unknown: here, a codec named
# `acme.codec` that takes no configuration.
MY_CODEC = CodecDefinition(
name="acme.codec", configuration=EmptyConfiguration, kind="bytes_bytes", size="dynamic"
MY_CODEC = BytesBytesCodecDefinition(
name="acme.codec", configuration=EmptyConfiguration, size="dynamic"
)
context = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[MY_CODEC]))
reading = read_array_metadata_v3(raw, context=context)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@

resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}},
CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)
resolved # RefusedDeclaration(..., definition=CodecDefinition(name='gzip'), ...)
resolved # RefusedDeclaration(..., definition=BytesBytesCodecDefinition(name='gzip'), ...)
problems[0].loc # ('configuration', 'level')
problems[0].input # 12
dict(problems[0].ctx) # {'ge': 0, 'le': 9}
Expand Down Expand Up @@ -125,7 +125,7 @@

from zarr_metadata import ValidationProblem
from zarr_metadata.definition import Nested
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, BytesBytesCodecDefinition, ZarrV3Context


class AcmeLz4Configuration(TypedDict, closed=True):
Expand All @@ -143,10 +143,9 @@ def acme_lz4_rules(
)


ACME_LZ4 = CodecDefinition(
ACME_LZ4 = BytesBytesCodecDefinition(
name="acme.lz4",
configuration=AcmeLz4Configuration,
kind="bytes_bytes",
size="dynamic",
rules=acme_lz4_rules,
)
Expand Down
66 changes: 30 additions & 36 deletions packages/zarr-metadata/src/zarr_metadata/testing/_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
from zarr_metadata._common import EmptyConfiguration
from zarr_metadata.definition._definition import AcceptedDeclaration
from zarr_metadata.testing._data_type import (
COMPLEX_WIDTHS,
FLOAT_WIDTHS,
INTEGER_RANGES,
float_fill_values,
Expand All @@ -35,10 +34,11 @@
fill_values_of,
pipelines_for,
read,
scope_of,
)
from zarr_metadata.testing._strategy import EMPTY_CONFIGURATIONS, MAX_RANK, CodecStrategy
from zarr_metadata.v3._definition import Chunk, DataTypeDefinition, storage_of
from zarr_metadata.v3.codec._arithmetic import FLOATING_POINT, NO_REAL_NUMBERS, NOT_NUMBERS
from zarr_metadata.v3.codec._arithmetic import NO_REAL_NUMBERS
from zarr_metadata.v3.codec.blosc import (
BLOSC_CNAME,
BLOSC_CODEC,
Expand Down Expand Up @@ -72,16 +72,20 @@
ZSTD_MIN_LEVEL,
ZstdCodecConfiguration,
)
from zarr_metadata.v3.data_type.numpy_datetime64 import NUMPY_DATETIME64_DATA_TYPE
from zarr_metadata.v3.data_type.numpy_timedelta64 import NUMPY_TIMEDELTA64_DATA_TYPE

if TYPE_CHECKING:
from collections.abc import Mapping

from zarr_metadata._common import JSONValue
from zarr_metadata.definition._definition import ResolvedDeclaration
from zarr_metadata.testing._strategy import Configurations, Strategies
from zarr_metadata.v3._definition import CodecDefinition, CodecSize, Lengths
from zarr_metadata.v3._definition import (
ArrayBytesCodecDefinition,
CodecDefinition,
CodecSize,
Lengths,
NumberClass,
)

_BLOSC_MAX_TYPESIZE: Final = 255
"""The widest element c-blosc takes: `BLOSC_MAX_TYPESIZE`."""
Expand Down Expand Up @@ -109,12 +113,6 @@
_SMALL_NUMBERS: Final = st.integers(0, 127)
"""Values every integral and floating-point data type holds: for a chunk whose data type is not known."""

_NUMPY_TIMES: Final = frozenset({NUMPY_DATETIME64_DATA_TYPE, NUMPY_TIMEDELTA64_DATA_TYPE})
"""The numpy time types, each stored as a signed 64-bit integer."""

_NUMPY_TIME_RANGE: Final = (-(2**63) + 1, 2**63 - 1)
"""The signed 64-bit integers but the least, which numpy reads as `NaT`: the finite numpy times."""


@st.composite
def _endians(draw: st.DrawFn, *, required: bool) -> BytesCodecConfiguration:
Expand Down Expand Up @@ -183,18 +181,14 @@ def _offsets_and_scales(
def _scale_offset(
chunk: Chunk, _strategies: Strategies
) -> st.SearchStrategy[ScaleOffsetCodecConfiguration] | None:
"""An `offset` and a `scale`, each a value of the data type the codec is handed, the scale not zero; None for a data type whose values are not numbers.
"""An `offset` and a `scale`, each a value of the data type the codec is handed, the scale not zero; None for a data type whose values are not integers or floating-point numbers.

Each is a fill value of the chunk's data type
(https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/codecs/scale_offset/README.md?plain=1#L39-L43),
so a float's is in every spelling a fill value takes, a `NaN` and an
infinity among them, and a complex number's is a pair of them. The
scale is drawn non-zero in any spelling, so that decoding never divides
by zero; a scale small enough that decoding overflows is still drawn. A
numpy time is drawn as the signed 64-bit integer it is stored as, never
`NaT`, since it "is compatible with any codec that supports arrays of
signed 64-bit integers"
(https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/data-types/numpy.datetime64/README.md?plain=1#L120).
infinity among them. The scale is drawn non-zero in any spelling, so
that decoding never divides by zero; a scale small enough that decoding
overflows is still drawn.
A data type this package does not define, whose values it cannot
draw, takes only the configuration with neither, the identity.
Handed a chunk whose data type is unknown, the values are small
Expand All @@ -205,22 +199,15 @@ def _scale_offset(
if definition is None:
offset: st.SearchStrategy[JSONValue] = _SMALL_NUMBERS
scale: st.SearchStrategy[JSONValue] = st.integers(1, 127)
elif definition.name in NOT_NUMBERS:
elif definition.number_class in NO_REAL_NUMBERS:
return None
elif definition in INTEGER_RANGES or definition in _NUMPY_TIMES:
low, high = INTEGER_RANGES.get(definition, _NUMPY_TIME_RANGE)
elif definition in INTEGER_RANGES:
low, high = INTEGER_RANGES[definition]
offset = st.integers(low, high)
scale = st.integers(low, high).filter(lambda value: value != 0)
elif definition in FLOAT_WIDTHS:
offset = float_fill_values(FLOAT_WIDTHS[definition])
scale = offset.filter(lambda value: not is_zero_float(value))
elif definition in COMPLEX_WIDTHS:
part = float_fill_values(COMPLEX_WIDTHS[definition])
offset = st.tuples(part, part)
# Non-zero in either part is a non-zero complex number.
scale = st.tuples(part, part).filter(
lambda pair: not (is_zero_float(pair[0]) and is_zero_float(pair[1]))
)
else:
return st.builds(ScaleOffsetCodecConfiguration)
return _offsets_and_scales(offset, scale)
Expand All @@ -235,7 +222,7 @@ def _zstd(draw: st.DrawFn) -> ZstdCodecConfiguration:
return configuration


def _vlen(definition: CodecDefinition[Any]) -> CodecStrategy:
def _vlen(definition: ArrayBytesCodecDefinition[Any]) -> CodecStrategy:
"""A codec with nothing to configure that takes only some data types: none for a chunk of another, which its own chunk rules say."""
return CodecStrategy(
configurations=lambda chunk, _strategies: (
Expand All @@ -251,9 +238,11 @@ def _unchunked(configurations: Configurations) -> CodecStrategy:
return CodecStrategy(configurations=lambda _chunk, _strategies: configurations)


def _name_read(data_type: ResolvedDeclaration[DataTypeDefinition[Any]] | None) -> str | None:
"""The name of the definition that accepted `data_type`; None when none did."""
return data_type.definition.name if isinstance(data_type, AcceptedDeclaration) else None
def _number_class_read(
data_type: ResolvedDeclaration[DataTypeDefinition[Any]] | None,
) -> NumberClass | None:
"""The number class the definition that accepted `data_type` declares; None when none accepted it, or it declares none."""
return data_type.definition.number_class if isinstance(data_type, AcceptedDeclaration) else None


def _source_scalars(chunk: Chunk, strategies: Strategies) -> st.SearchStrategy[JSONValue]:
Expand Down Expand Up @@ -294,7 +283,7 @@ def _cast_value_configuration(
declaration, DataTypeDefinition, strategies
)
modes: tuple[CastOutOfRangeMode, ...] = CAST_OUT_OF_RANGE_MODE
if _name_read(target) in FLOATING_POINT:
if _number_class_read(target) == "floating_point":
modes = tuple(mode for mode in modes if mode != "wrap")
configuration = CastValueCodecConfiguration(data_type=declaration)
if draw(st.booleans()):
Expand All @@ -320,9 +309,14 @@ def _cast_value(
of its direction; `wrap` is drawn only for a type that is not
floating-point.
"""
if _name_read(chunk.data_type) in NO_REAL_NUMBERS:
if _number_class_read(chunk.data_type) in NO_REAL_NUMBERS:
return None
targets = [name for name in data_type_names(strategies) if name not in NO_REAL_NUMBERS]
data_types = scope_of(strategies).data_types
targets = [
name
for name in data_type_names(strategies)
if data_types[name].number_class not in NO_REAL_NUMBERS
]
if len(targets) == 0:
return None
return _cast_value_configuration(targets, _source_scalars(chunk, strategies), strategies)
Expand Down
39 changes: 25 additions & 14 deletions packages/zarr-metadata/src/zarr_metadata/testing/_draw.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from __future__ import annotations

from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final, cast
from typing import TYPE_CHECKING, Any, Final, TypeVar, cast

from hypothesis import reject
from hypothesis import strategies as st
Expand All @@ -21,7 +21,15 @@
from zarr_metadata.testing._strategy import NOTHING_KNOWN
from zarr_metadata.v3._context import ZarrV3Context
from zarr_metadata.v3._core import CORE_CONTEXT
from zarr_metadata.v3._definition import CodecDefinition, DataTypeDefinition, no_pipelines
from zarr_metadata.v3._definition import (
ArrayArrayCodecDefinition,
ArrayBytesCodecDefinition,
ArrayCodecDefinition,
BytesBytesCodecDefinition,
CodecDefinition,
DataTypeDefinition,
no_pipelines,
)

if TYPE_CHECKING:
from collections.abc import Iterable, Iterator, Sequence
Expand All @@ -39,7 +47,10 @@
)
from zarr_metadata.v3 import ZarrV3DeclarationJSON
from zarr_metadata.v3._context import ZarrV3ContextSlots
from zarr_metadata.v3._definition import Chunk, CodecKind, CodecSize
from zarr_metadata.v3._definition import Chunk, CodecSize

K = TypeVar("K", bound="CodecDefinition[Any]")
"""A kind of codec: the class of its definitions."""

_MAX_ARRAY_ARRAY: Final = 2
"""The most array -> array codecs a pipeline is drawn with."""
Expand Down Expand Up @@ -221,20 +232,20 @@ def fill_values_of(

def _taking(
chunk: Chunk,
kind: CodecKind,
kind: type[K],
sizes: frozenset[CodecSize],
strategies: Strategies,
*,
holding: bool = True,
) -> list[tuple[CodecDefinition[Any], Configurations]]:
) -> list[tuple[K, Configurations]]:
"""Each codec of `kind` and one of `sizes` that `strategies` draw for, that takes `chunk`, with its configurations.

Without `holding`, a codec that holds pipelines of its own is left
out: what one of those takes depends on what its pipelines do.
"""
taking: list[tuple[CodecDefinition[Any], Configurations]] = []
taking: list[tuple[K, Configurations]] = []
for definition in sorted(scope_of(strategies).codecs.values(), key=lambda d: d.name):
if definition.kind != kind or definition.size not in sizes:
if not isinstance(definition, kind) or definition.size not in sizes:
continue
if not holding and _holds_pipelines(definition):
continue
Expand All @@ -246,7 +257,7 @@ def _taking(


def _holds_pipelines(definition: CodecDefinition[Any]) -> bool:
return definition.pipelines is not no_pipelines
return isinstance(definition, ArrayCodecDefinition) and definition.pipelines is not no_pipelines


def pipelines_for(
Expand All @@ -258,15 +269,15 @@ def pipelines_for(
that holds pipelines of its own takes a chunk only if what it holds
does, so it settles nothing.
"""
if len(_taking(chunk, "array_bytes", sizes, strategies, holding=False)) == 0:
if len(_taking(chunk, ArrayBytesCodecDefinition, sizes, strategies, holding=False)) == 0:
return None
return _walk(chunk, strategies, sizes)


@st.composite
def _codec(
draw: st.DrawFn, options: Sequence[tuple[CodecDefinition[Any], Configurations]]
) -> tuple[CodecDefinition[Any], ZarrV3DeclarationJSON]:
draw: st.DrawFn, options: Sequence[tuple[K, Configurations]]
) -> tuple[K, ZarrV3DeclarationJSON]:
definition, configurations = draw(st.sampled_from(options))
return definition, draw(any_spelling(definition, configurations))

Expand All @@ -285,19 +296,19 @@ def _walk(
codecs: list[ZarrV3DeclarationJSON] = []
handed = chunk
for _ in range(draw(st.integers(0, _MAX_ARRAY_ARRAY))):
options = _taking(handed, "array_array", sizes, strategies)
options = _taking(handed, ArrayArrayCodecDefinition, sizes, strategies)
if len(options) == 0:
break
definition, declaration = draw(_codec(options))
codec = read(declaration, CodecDefinition, strategies)
assert isinstance(codec, AcceptedDeclaration)
handed = definition.transition(codec.configuration, codec.nested, handed)
codecs.append(declaration)
options = _taking(handed, "array_bytes", sizes, strategies)
options = _taking(handed, ArrayBytesCodecDefinition, sizes, strategies)
if len(options) == 0:
reject() # what the array -> array codecs handed on, no array -> bytes codec takes
codecs.append(draw(_codec(options))[1])
options = _taking(NOTHING_KNOWN, "bytes_bytes", sizes, strategies)
options = _taking(NOTHING_KNOWN, BytesBytesCodecDefinition, sizes, strategies)
if len(options) != 0:
count = draw(st.integers(0, _MAX_BYTES_BYTES))
codecs.extend(draw(_codec(options))[1] for _ in range(count))
Expand Down
Loading
Loading