Repository navigation
fix: preserve lambda bindings in nested-loop join projection pushdown - #26399
Merged
xudong963 merged 2 commits intoOct 19, 2026
Merged
Conversation
Keep lambda bodies within their high-order functions while retaining single-input expression pushdown. Rebase lambda parameters as expressions move between input projections and join filter schemas, with SQL result and plan regressions. Generated-by: OpenAI Codex
github-actions
Bot
added
optimizer
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26399 +/- ##
=======================================
Coverage 82.89% 82.89%
=======================================
Files 1154 1154
Lines 458022 458058 +36
Branches 458022 458058 +36
=======================================
+ Hits 379683 379720 +37
+ Misses 55303 55298 -5
- Partials 23036 23040 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
I think this issue is related. #24372 |
jayzhan211
approved these changes
Oct 18, 2026
jayzhan211
left a comment
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @xudong963 , LGTM!
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 19, 2026
mkleen
pushed a commit
to mkleen/datafusion
that referenced
this pull request
Oct 20, 2026
…apache#26404) ## Which issue does this PR close? - Closes apache#26349. ## Rationale for this change `AggregateStatistics` can answer a scalar subquery such as `(SELECT min(ts) FROM live)` from statistics and replace it with a row of literals. A join against that row still runs as a `NestedLoopJoinExec`, or as a `CrossJoinExec` when the join has no condition. That join only filters its other input, but neither join operator preserves its inputs' ordering. Operators above it therefore lose the input's ordering. For example, an aggregation grouped by the input's sort key can no longer emit each group as soon as its sort key changes (`group_clustering_mode=Full`) and must hold every group until the end. The literal bound is also not pushed into the scan for pruning. ## What changes are included in this PR? A new physical optimizer rule, `EliminateConstantJoin`, runs right after `AggregateStatistics`. It applies when one input of an inner `NestedLoopJoinExec` or a `CrossJoinExec` is provably a single row of literals: projections over a single-partition `PlaceholderRowExec`. The rule substitutes the literals into the join filter. It then replaces the join with a `FilterExec` over the other input, under a `ProjectionExec` that restores the join's output schema and metadata. A cross join has no filter, so it becomes only the projection. Besides dropping ordering, `CrossJoinExec` buffers its other input as the build side when that input has no statistics. At this point in the optimizer, a folded `min(ts) AS first_ts` is still an alias projection over the literal projection, so the rule follows stacked projections. Expressions over a folded value, such as `min(ts) + 1`, only become constant at this point, so the rule folds them with `PhysicalExprSimplifier`. The rule leaves the join unchanged when: - the join is not an inner join; - the placeholder has more than one partition, because it emits one row per partition; - a value does not fold to a literal; for example, `random()` would then be evaluated per row instead of once; - the join filter contains a lambda, because lambda variables are numbered after the filter's own columns, not the other input's. ## What is the testing strategy for this PR? - `eliminate_constant_join.slt` has plans and results for four rewrites: a scalar folded from Parquet statistics, an expression over it (`min(ts) + 1`), a cross join, and a literal subquery. In the first and the cross join, the aggregation becomes `group_clustering_mode=Full`; the bound is pushed into the Parquet scan. The file also checks that a left join and a `random()` constant keep the join. - `physical_optimizer/eliminate_constant_join.rs` covers cases SQL cannot reach: - a multi-partition placeholder; - a lambda join filter, planned without optimizer rules because projection pushdown cannot plan it yet (apache#26399); - schema metadata that comes from the constant input. - `explain.slt`: `EXPLAIN VERBOSE` lists the new rule. - `cte.slt`, `joins.slt`: `j1, LATERAL (SELECT 1)` now plans a projection instead of a `CrossJoinExec`. ## Are there any user-facing changes? Queries that join against a row folded from statistics, or against a literal subquery, now plan a filter and a projection instead of a nested loop join or cross join. `EXPLAIN VERBOSE` and `PhysicalOptimizer::new()` include the new rule. There are no API changes.
xudong963
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Oct 22, 2026
…apache#26404) ## Which issue does this PR close? - Closes apache#26349. ## Rationale for this change `AggregateStatistics` can answer a scalar subquery such as `(SELECT min(ts) FROM live)` from statistics and replace it with a row of literals. A join against that row still runs as a `NestedLoopJoinExec`, or as a `CrossJoinExec` when the join has no condition. That join only filters its other input, but neither join operator preserves its inputs' ordering. Operators above it therefore lose the input's ordering. For example, an aggregation grouped by the input's sort key can no longer emit each group as soon as its sort key changes (`group_clustering_mode=Full`) and must hold every group until the end. The literal bound is also not pushed into the scan for pruning. ## What changes are included in this PR? A new physical optimizer rule, `EliminateConstantJoin`, runs right after `AggregateStatistics`. It applies when one input of an inner `NestedLoopJoinExec` or a `CrossJoinExec` is provably a single row of literals: projections over a single-partition `PlaceholderRowExec`. The rule substitutes the literals into the join filter. It then replaces the join with a `FilterExec` over the other input, under a `ProjectionExec` that restores the join's output schema and metadata. A cross join has no filter, so it becomes only the projection. Besides dropping ordering, `CrossJoinExec` buffers its other input as the build side when that input has no statistics. At this point in the optimizer, a folded `min(ts) AS first_ts` is still an alias projection over the literal projection, so the rule follows stacked projections. Expressions over a folded value, such as `min(ts) + 1`, only become constant at this point, so the rule folds them with `PhysicalExprSimplifier`. The rule leaves the join unchanged when: - the join is not an inner join; - the placeholder has more than one partition, because it emits one row per partition; - a value does not fold to a literal; for example, `random()` would then be evaluated per row instead of once; - the join filter contains a lambda, because lambda variables are numbered after the filter's own columns, not the other input's. ## What is the testing strategy for this PR? - `eliminate_constant_join.slt` has plans and results for four rewrites: a scalar folded from Parquet statistics, an expression over it (`min(ts) + 1`), a cross join, and a literal subquery. In the first and the cross join, the aggregation becomes `group_clustering_mode=Full`; the bound is pushed into the Parquet scan. The file also checks that a left join and a `random()` constant keep the join. - `physical_optimizer/eliminate_constant_join.rs` covers cases SQL cannot reach: - a multi-partition placeholder; - a lambda join filter, planned without optimizer rules because projection pushdown cannot plan it yet (apache#26399); - schema metadata that comes from the constant input. - `explain.slt`: `EXPLAIN VERBOSE` lists the new rule. - `cte.slt`, `joins.slt`: `j1, LATERAL (SELECT 1)` now plans a projection instead of a `CrossJoinExec`. ## Are there any user-facing changes? Queries that join against a row folded from statistics, or against a literal subquery, now plan a filter and a projection instead of a nested loop join or cross join. `EXPLAIN VERBOSE` and `PhysicalOptimizer::new()` include the new rule. There are no API changes. (cherry picked from commit 2c2ecae) Adapt SQL plan expectations to branch-55, which still uses ordering_mode=Sorted instead of group_clustering_mode=Full.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
A nested-loop join whose lambda captures a column from the other input can fail during physical optimization with
unable to unwrap lambda, even though the query executes correctly with physical optimization disabled.These queries should work with the default optimizer while retaining projection pushdown for expressions that can be evaluated on one join input.
What changes are included in this PR?
nested_loop_join_lambda.sltwith result and physical-plan regressions.This change preserves lambda bodies within their enclosing function; it does not introduce hoisting of individual computations from inside a lambda body.
What is the testing strategy for this PR?
The new SQL test covers the original reproducer with either join-input order, mixed predicates, nested lambdas, parameter shadowing, empty/NULL arrays, and extraction of complete high-order functions onto either input. Different input and filter schema widths exercise parameter rebasing. Three
EXPLAINchecks verify that ordinary expressions and complete high-order functions are still pushed down.Are there any user-facing changes?
Nested-loop joins with lambda predicates now plan and execute with the default physical optimizer. There are no public API or configuration changes.