Repository navigation
fix: [branch-1.2] preserve columnar output when reverting a cached AQE result stage (#6852) - #7650
Merged
Merged
Conversation
apache#6852) * fix: preserve columnar output when reverting a cached AQE result stage RevertNativeForTransitionHeavyStages passed outputColumnar = false for every result stage. On Spark 4.0+, when the cache serializer accepts columnar input, as Comet's does, Spark marks a cached AQE plan columnar, AQE plans its final stage to output columnar batches, and the cache builder executes it columnar. A reverted row-based root such as FilterExec then failed with "FilterExec has column support mismatch". Read the requested format off the stage root instead. Spark's transition insertion leaves a columnar-only root only when columnar output was requested, so a row-based reverted root gets a RowToColumnarExec, as Spark itself would insert. * fix: skip reverting a result stage whose root supports both formats unless the reverted root does too * fix: apply the dual-format result-stage check only under AQE Without AQE the consumer of a result stage reads rows, or, for a cache, picks the format from the root this rule leaves, so a root that supports both formats goes back to row output as it does on main. Only AQE fixes the requested format before it plans the final stage. The check reads the AQE conf rather than wholePlan, so the plan-only preview under AQE keeps the same stages that AQE does. * test: cover the result root's output format at the rule level on every Spark version The cache test that exercises restoring columnar output for a columnar-only root is skipped before Spark 4.0, although the branch it covers does not depend on the Spark version. Apply the rule to a columnar-only root and to a row root directly, and assert the format each keeps. (cherry picked from commit d7f4fca)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #6852 to
branch-1.2. Closes #6687 on this branch.Rationale for this change
RevertNativeForTransitionHeavyStagesassumed a result stage is always collected as rows. On Spark 4.0+, a cached query whose serializer accepts columnar input, as Comet'sArrowCachedBatchSerializerdoes, gets a columnar result stage, and once the rule reverted that stage to a row-based root nothing restored the columnar output. Building the cache then failed withFilterExec has column support mismatch(#6687).branch-1.2has the same rule and the same call sites. Transition reversion is off by default, so only users who enablespark.comet.exec.transitionRevert.enabledare affected. See #6852 for the full analysis.What changes are included in this PR?
Cherry-pick of d7f4fca (
git cherry-pick -x) with no conflicts and no adaptations. The diff against the source commit is identical apart from line numbers and blob hashes. Both result-stage call sites inRevertNativeForTransitionHeavyStagesgo through a newrevertResultStageIfNeeded, which reads the requested output format off the stage root, andRevertNativeForTransitionHeavyStagesSuitegets the source PR's tests.Other release branches:
branch-1.2is the newest release branch, so there is no newer one to port to. Whetherbranch-1.1needs a backport is not part of this PR.How are these changes tested?
On this branch, with JDK 17 and the default Spark profile (4.1):
./mvnw test-compilesucceeds, andRevertNativeForTransitionHeavyStagesSuitepasses (43 tests, 0 failures). No native source changed. Withspark/src/mainreverted tobranch-1.2's version, five of the new tests fail:transition-heavy revert restores columnar output for a cached AQE result stage, the twoAQE=truecases oftransition-heavy revert of a result root that supports both formats, and bothtransition-heavy revert keeps the output format of the result rootcases. So the branch has the bug and the tests catch it. I did not run the other Spark profiles.