Skip to content

feat(datafusion): support lateral collection generators - #1292

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/fusion-lateral-views
Nov 2, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/fusion-lateral-views

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes #1284.

DataFusion rejects LATERAL VIEW. A correlated UNNEST subquery is insufficient because the native physical planner cannot execute the projected outer reference.

Brief change log

  • Support explode/posexplode and outer variants for arrays/maps with zero-based positions and exact empty/NULL behavior.
  • Use a relation-planner extension to plan the parent input and native list/struct UNNEST without correlated physical expressions or collecting the parent.
  • Preserve aliases, joins, chained generators, set-operation branches and portable persistent-view SQL; compensate for the pinned parser's optional-alias clause handling.

Tests

  • Retain a permanent integration regression that uses comment DDL, binary casts, regex operators, lateral generators, scalar functions and constant CALL arguments together, protecting composition of their shared SQL hooks.

  • Regression coverage: Array/Map values, NULL and empty collections, NULL elements, empty parent inputs, map defaults, quoted and explicit aliases, joins/duplicate rows, chained generators, UNION, dictionary collection/value nulls, and default-alias ORDER/LIMIT/OFFSET/QUALIFY boundaries.

  • Reproduced the missing behavior on the original main before implementation.

  • Passed cargo test --locked -p paimon-datafusion --all-targets -- --test-threads=4 on this independent branch with Rust 1.95.0, TZ=UTC and RUST_MIN_STACK=16777216.

  • Passed cargo +1.98.0 clippy --locked --no-deps -p paimon-datafusion --all-targets -- -D warnings, formatting and git diff --check.

  • Reviewed repeatedly for SQL semantics, parser/type boundaries and regression quality; fixed the discovered alias, set-operation and default-column parsing cases.

  • Integration validation: combined with feat(datafusion): support table and column comment DDL #1287, feat(datafusion): support binary casts and byte widths #1288, feat(datafusion): support RLIKE and REGEXP operators #1289, feat(datafusion): add common scalar compatibility functions #1291, feat(datafusion): evaluate immutable constant CALL arguments #1290 in a temporary checkout; the full DataFusion suite, a regression using all six extensions together, Clippy and formatting passed. These PRs are independently based on main; overlapping parser/registration hooks need normal conflict resolution when merged sequentially.

API and Format

Adds SQLContext SQL capabilities. No storage-format or Rust public API changes.

Documentation

Documented syntax, type/null/error behavior and supported limits in docs/src/sql.md.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit cccc24d into apache:main Nov 2, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[datafusion] Support lateral array and map generators

2 participants