You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Memoize LAST_VALUE and NTH_VALUE with IGNORE NULLS so BoundedWindowAggExec can prune its buffer #26537
Is your feature request related to a problem or challenge?
NthValueEvaluator::memoize returns early for LAST_VALUE and NTH_VALUE when IGNORE NULLS is set (nth_value.rs#L414-L415). For a frame with an UNBOUNDED PRECEDING start, such as the default RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, the frame start then never moves. Reading the code, this means BoundedWindowAggExec cannot prune its input buffer and keeps the whole partition in memory, while RESPECT NULLS keeps only the last row (LAST_VALUE) or the last n rows (NTH_VALUE(expr, -n)).
#26533 fixes the CPU cost of these frames (#26532), but not the buffering. I have not measured the memory use.
Describe the solution you'd like
Let memoize move the frame start under IGNORE NULLS too, keeping the rows the result can still depend on:
LAST_VALUE: from the last non-NULL row of the frame, or nothing before the current row if the frame has none.
NTH_VALUE(expr, -n): from the nth non-NULL row from the end of the frame.
NTH_VALUE(expr, n) with n > 0: the result is final once the frame holds n non-NULL rows, as FIRST_VALUE already does.
Activity