Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 10 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ The specification version a release targets is set by the `substrait-protobuf`,
definitions and the grammar all come from those packages rather than being vendored
here. `substrait.version.substrait_version` reports it at runtime.

When upgrading from spec v0.106.0 to v0.108.0, pass an IANA time zone when subtracting an `interval_year` from a `precision_timestamp_tz<P>`. For example, use `sub.f.subtract(sub.col("ts"), sub.col("offset"), "America/New_York")` instead of the two-argument call.

# Building plans with the DataFrame API

The `substrait.dataframe` module is an ergonomic, fluent API for authoring
Expand Down Expand Up @@ -66,6 +68,8 @@ The examples below construct plans with the raw `substrait.proto` and
`substrait.builders` layers. For most hand-authored plans, prefer the
[DataFrame API](#building-plans-with-the-dataframe-api) above.

When combining plans with named lambdas, give different definitions different `lambda_anchor` values. The builders preserve these anchors and reject conflicting definitions rather than silently changing which lambda a call uses. An `ExtendedExpression` cannot carry named lambda definitions. If a subquery uses a named lambda, or an opaque extension payload might refer to one, the builder rejects the subquery instead of discarding its definitions.

## Produce a Substrait Plan
The ``substrait.proto`` module provides access to the classes
that represent a substrait Plan, thus allowing to create new plans.
Expand Down Expand Up @@ -252,28 +256,20 @@ relations {
```

## Deeply nested expressions
Protobuf parsers, including the one in the Python `protobuf` library, reject messages nested more
than 100 levels deep, and each expression level takes several message levels. Substrait works
around this with [detached expressions](https://substrait.io/serialization/binary_serialization/#bounded-expression-nesting):
deep subtrees are moved to a table on the `PlanRel` or `ExtendedExpression` and referenced by
ordinal.

Protobuf parsers limit how deeply a message can nest. To stay within that limit, a producer can store an expression or relation subtree in a table on the containing `PlanRel` or `ExtendedExpression`. An ordinal in the tree refers to the table entry. See [detached relations and expressions](https://substrait.io/serialization/binary_serialization/#detached-relations-and-expressions).

```python
from substrait.proto import Plan
from substrait.utils.detached import detach, resolve_detached

data = detach(plan).SerializeToString() # parses under the default limits
plan = resolve_detached(Plan.FromString(data)) # ordinary expression trees again
data = detach(plan).SerializeToString()
plan = resolve_detached(Plan.FromString(data))
```

`detach` returns a plan with nothing to detach as is. `infer_plan_schema`,
`infer_extended_expression_schema`, the plan printer, and the builders resolve detached expressions
on their own. Functions that take a bare `Rel` or `Expression` cannot, since the table lives on the
containing message, so resolve the `Plan` or `ExtendedExpression` first.
`detach` splits deep expressions when writing a plan. It does not split deep relation trees. When reading, `resolve_detached` inlines both relation and expression references. Schema inference, the plan printer, and the builders do this automatically. If a function takes a bare `Rel` or `Expression`, resolve the containing message first so the tables are available.

The builders work on the inlined tree, and the `protobuf` library copies messages passed to a
constructor through its parser, so building on a detached plan works only while the plan still fits
the limit once inlined. Detach the finished plan instead, just before serializing it.
The builders work on the inlined tree. The `protobuf` library copies messages passed to a constructor through its parser, so the inlined tree must still fit the parser's limit. When building a plan, call `detach` just before serialization.

## Produce a Substrait Plan with Ibis
Let's use an existing Substrait producer, [Ibis](https://ibis-project.org),
Expand Down
147 changes: 73 additions & 74 deletions pixi.lock

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,20 @@ readme = "README.md"
requires-python = ">=3.10,<3.15"
dependencies = [
"protobuf >=5,<8",
"substrait-protobuf==0.106.0",
"substrait-extensions==0.106.0",
"substrait-protobuf==0.108.0",
"substrait-extensions==0.108.0",
]
dynamic = ["version"]

[tool.setuptools_scm]
write_to = "src/substrait/_version.py"

[project.optional-dependencies]
extensions = ["substrait-antlr==0.106.0", "pyyaml"]
extensions = ["substrait-antlr==0.108.0", "pyyaml"]
sql = ["sqloxide", "deepdiff"]

[dependency-groups]
dev = ["pytest >= 7.0.0", "substrait-antlr==0.106.0", "pyyaml", "sqloxide", "deepdiff", "duckdb>=1.5,<1.6", "datafusion"]
dev = ["pytest >= 7.0.0", "substrait-antlr==0.108.0", "pyyaml", "sqloxide", "deepdiff", "duckdb>=1.5,<1.6", "datafusion"]

[tool.pytest.ini_options]
pythonpath = "src"
Expand Down
12 changes: 12 additions & 0 deletions src/substrait/builders/extended_expression.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import substrait.algebra_pb2 as stalg
import substrait.extended_expression_pb2 as stee
import substrait.type_pb2 as stp
from google.protobuf.any_pb2 import Any as ProtoAny

from substrait.extension_registry import (
ExtensionRegistry,
Expand All @@ -21,6 +22,7 @@
from substrait.type_inference import infer_extended_expression_schema, outer_schemas
from substrait.utils import (
inline_reference_rels,
iter_proto_messages,
plan_subtrees,
remap_function_references,
top_level_names,
Expand Down Expand Up @@ -1106,6 +1108,16 @@ def _inner_rel(query, registry: ExtensionRegistry, base_schema) -> stalg.Rel:
subtrees = plan_subtrees(plan)
if subtrees:
rel = inline_reference_rels(rel, subtrees)
if plan.named_lambdas and any(
isinstance(
msg, (stalg.Expression.NamedLambdaReference, stalg.ReferenceRel, ProtoAny)
)
for msg in iter_proto_messages(rel)
):
raise ValueError(
"cannot embed a subquery that may reference a named lambda: "
"its definitions do not travel into the ExtendedExpression"
)
return rel


Expand Down
102 changes: 97 additions & 5 deletions src/substrait/builders/plan.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import substrait.extended_expression_pb2 as stee
import substrait.plan_pb2 as stp
import substrait.type_pb2 as stt
from google.protobuf.any_pb2 import Any as ProtoAny
from substrait.extensions.extensions_pb2 import AdvancedExtension

from substrait.builders.extended_expression import (
Expand All @@ -35,10 +36,13 @@
)
from substrait.utils import (
child_rels,
iter_proto_messages,
named_lambda_bindings,
plan_subtrees,
rebase_reference_ordinals,
remap_function_references,
type_num_names,
validate_named_lambda_references,
)
from substrait.utils.detached import resolve_for_building
from substrait.version import substrait_version
Expand Down Expand Up @@ -82,7 +86,56 @@ def _bind(plan: PlanOrUnbound, registry: ExtensionRegistry) -> stp.Plan:
collector = current_collector()
if collector is None:
return bound
return remap_function_references(bound, collector.adopt(bound))
remap = collector.adopt(bound)
if (
bound.named_lambdas
and remap
and any(isinstance(msg, ProtoAny) for _, msg in _named_lambda_messages(bound))
):
raise ValueError("cannot remap function anchors inside an opaque named lambda")
return remap_function_references(bound, remap)


def _named_lambda_messages(plan):
"""Walk definitions and the shared subtrees they reference, visiting each subtree once."""
subtrees = dict(enumerate(plan_subtrees(plan)))
pending = [
(definition.lambda_anchor, definition) for definition in plan.named_lambdas
]
while pending:
anchor, root = pending.pop()
for msg in iter_proto_messages(root):
yield anchor, msg
if isinstance(msg, stalg.ReferenceRel):
subtree = subtrees.pop(msg.subtree_ordinal, None)
if subtree is not None:
pending.append((anchor, subtree))


def _check_lambda_metadata(plan):
"""Reject definitions that could depend on metadata a new Plan does not copy."""
if not plan.type_aliases and not plan.parameter_bindings:
return
bound_parameters = {binding.parameter_anchor for binding in plan.parameter_bindings}
for anchor, msg in _named_lambda_messages(plan):
if isinstance(msg, ProtoAny):
field = "type_aliases or parameter_bindings"
elif isinstance(msg, stt.Type.TypeAliasReference) or (
isinstance(msg, stalg.Expression.Literal.UserDefined)
and msg.HasField("type_alias_reference")
):
field = "type_aliases"
elif (
isinstance(msg, stalg.DynamicParameter)
and msg.parameter_reference in bound_parameters
):
field = "parameter_bindings"
else:
continue
raise ValueError(
f"cannot preserve named lambda {anchor}: "
f"the builder does not copy {field} that it may reference"
)


def _merge_plan_metadata(*objs):
Expand All @@ -98,12 +151,37 @@ def _merge_plan_metadata(*objs):
Extension URNs and declarations are *not* merged here: they belong to the
build's ``ExtensionCollector``, which writes them onto the outermost plan once
(see ``build_scoped``), rather than being re-merged at every level.

Named lambda anchors keep their original numbers. Two inputs can share an
anchor if it denotes the same lambda after extension function remapping.
In that case, keep the first diagnostic name. When an input gains bindings,
check its original scope first so another input cannot satisfy a missing reference.
"""
metadata = {}
for b in objs:
if isinstance(b, stp.Plan) and b.HasField("execution_behavior"):
metadata["execution_behavior"] = b.execution_behavior
break
bindings = {}
for source in objs:
if not isinstance(source, stp.Plan):
continue
if source.HasField("execution_behavior"):
metadata.setdefault("execution_behavior", source.execution_behavior)
if source.named_lambdas:
_check_lambda_metadata(source)
for anchor, definition in named_lambda_bindings(source).items():
previous = bindings.setdefault(anchor, definition)
if previous is not definition and getattr(
previous, "lambda"
) != getattr(definition, "lambda"):
raise ValueError(
f"conflicting definitions for named lambda anchor {anchor}"
)
if bindings:
for source in objs:
if source is not None and (
not isinstance(source, stp.Plan)
or len(source.named_lambdas) < len(bindings)
):
validate_named_lambda_references(source)
metadata["named_lambdas"] = list(bindings.values())
return metadata


Expand Down Expand Up @@ -144,6 +222,20 @@ def _merge_input_subtrees(bound_inputs):
key_to_ordinal[key] = new_ordinal
combined.append(rebased)
remap[old_ordinal] = new_ordinal
if (
plan.named_lambdas
and not _is_identity(remap)
and any(
isinstance(msg, ProtoAny)
or isinstance(msg, stalg.ReferenceRel)
and remap.get(msg.subtree_ordinal, msg.subtree_ordinal)
!= msg.subtree_ordinal
for _, msg in _named_lambda_messages(plan)
)
):
raise ValueError(
"cannot rebase shared subtree references inside a named lambda"
)
root_input = plan.relations[-1].root.input
rebased_root_inputs.append(
root_input
Expand Down
3 changes: 2 additions & 1 deletion src/substrait/dataframe/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
from substrait.extension_registry import ExtensionRegistry, build_scoped
from substrait.type_inference import infer_plan_schema
from substrait.utils import to_id_based_outer_references, top_level_names
from substrait.utils.detached import resolve_present_tables

# All 13 JoinRel.JoinType variants (SET_OP_UNSPECIFIED excluded). "single"
# returns at most one right match per left row (runtime error on multiple);
Expand Down Expand Up @@ -626,7 +627,7 @@ def hint(
inner = self._plan

def resolve(registry: ExtensionRegistry):
bound = inner(registry)
bound = resolve_present_tables(inner(registry))
rel = bound.relations[-1].root.input
rel_inner = getattr(rel, rel.WhichOneof("rel_type"))
# A ReferenceRel (from .cache()) carries no RelCommon and so cannot
Expand Down
Loading
Loading