Skip to content

fix: [branch-1.2] fall back to Spark for a nondeterministic child of size, arrays_zip and map_from_arrays - #7670

Draft
dwsmith1983 wants to merge 1 commit into
apache:branch-1.2from
dwsmith1983:fix/5781-nondeterministic-fallback-branch-1.2
Draft

dwsmith1983 wants to merge 1 commit into
apache:branch-1.2from
dwsmith1983:fix/5781-nondeterministic-fallback-branch-1.2

Conversation

@dwsmith1983

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #5781 on branch-1.2. #5867 fixes it on main and closes the issue; it is approved but not merged yet, so this should merge after it.

Rationale for this change

size, arrays_zip and map_from_arrays evaluate a nondeterministic child twice in 1.2.0: once as the value and again inside the native NULL guard. A stateful child such as IF(monotonically_increasing_id() % 2 = 0, array(1), NULL) then returns wrong results.

#5867 routes these cases through the JVM codegen dispatcher, but it also moves map lookups with float, collated and complex keys to the dispatcher (#5580), an enhancement that #5875 measured at 9 to 11 percent slower for lookup-only DOUBLE-key projections. That does not belong in a patch release, so this change falls back to Spark instead, as branch-1.2 already does for array_append and ANSI element_at.

What changes are included in this PR?

  • size and arrays_zip report a nondeterministic child as unsupported, which falls back to Spark.
  • map_from_arrays declines a nondeterministic child when it is converted, since reporting it unsupported would send it to the codegen dispatcher. This also covers the allowIncompatible route for float keys.
  • The fallback reason uses the same text as fix: dispatch map lookups with normalized keys and nondeterministic null-guarded children #5867.
  • The compatibility pages do not list the map_from_arrays fallback, because the generator would describe that function's unsupported reasons as running through the codegen dispatcher.
  • The compatibility pages list the new reason under ArraysZip and a new Size section. The pages for Spark 3.5, 4.0, 4.1 and 4.2 were regenerated; the Spark 3.4 page got the same lines by hand.

A non-nullable nondeterministic child, such as size(array(monotonically_increasing_id())), also falls back now. 1.2.0 ran it natively and correctly, since the guard passes every row.

How are these changes tested?

Three SQL file tests, adapted from #5867, check each function against Spark: a nondeterministic nullable child falls back, a non-nullable one falls back, and a deterministic or NULL input still runs natively. For map_from_arrays, DOUBLE keys go to the dispatcher by default and fall back with allowIncompatible=true. On branch-1.2 without the fix, all four nondeterministic cases and the allowIncompatible case return wrong results. With it, the tests pass on Spark 3.5, 4.0, 4.1 and 4.2, and the existing size, arrays_zip and map_from_arrays tests and the array and map expression suites still pass on 4.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant