Skip to content

fix: correct NaN grouping and eviction in TopK aggregation - #26445

Open
bvolpato wants to merge 2 commits into
apache:mainfrom
bvolpato:bvolpato/fix-topk-nan-grouping
Open

bvolpato wants to merge 2 commits into
apache:mainfrom
bvolpato:bvolpato/fix-topk-nan-grouping

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #26444.

Rationale for this change

Repeated identical NaN keys can form separate TopK groups, and evicting a NaN group can panic. Valid GROUP BY ... ORDER BY ... LIMIT queries can therefore return incorrect results or fail.

What changes are included in this PR?

  • Compare primitive grouping keys with Arrow equality so identical NaN payloads match.
  • Normalize signed zero consistently with ordinary grouping while preserving distinct NaN payloads.
  • Remove occupied entries by their known store index during eviction and NULL aggregate transitions.

A separate prerequisite commit retains the RSS sampling fix from #26208. It samples the current process before query execution so short queries cannot leave the recorded maximum at zero. Memory thresholds are unchanged.

What is the testing strategy for this PR?

The SQL regressions cover repeated NaN keys and eviction. Applied to current main (7f493a565), the eviction regression fails with internal error: entered unreachable code; it passes with this fix. Hash-table tests cover Float16/32/64, distinct NaN payloads, signed zero, and NULL aggregate transitions.

The extended workspace suite, full SQL logic suite, and cargo test --profile ci -p datafusion-cli passed. cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and uv run ./dev/rust_lint.sh also passed.

The existing topk_aggregate primitive-key case showed no significant change against main. This comparison used the unoptimized CI profile on a shared host, so its performance evidence is limited.

Are there any user-facing changes?

Identical NaN keys form one TopK group, and evicting a NaN group succeeds. Signed-zero grouping matches ordinary aggregation. There are no public API changes.

AI-assisted contribution.

@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Oct 17, 2026
@codecov-commenter

codecov-commenter commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.54930% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.93%. Comparing base (7f493a5) to head (fdd9fd1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...on/physical-plan/src/aggregates/topk/hash_table.rs 91.54% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #26445   +/-   ##
=======================================
  Coverage   82.93%   82.93%           
=======================================
  Files        1154     1154           
  Lines      457851   457915   +64     
  Branches   457851   457915   +64     
=======================================
+ Hits       379701   379761   +60     
  Misses      55131    55131           
- Partials    23019    23023    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bvolpato
bvolpato marked this pull request as ready for review October 17, 2026 14:55
@bvolpato bvolpato changed the title fix: handle NaN grouping keys in TopK aggregation fix: correct NaN grouping and eviction in TopK aggregation Oct 18, 2026
@bvolpato
bvolpato force-pushed the bvolpato/fix-topk-nan-grouping branch from 98d4394 to fdd9fd1 Compare October 20, 2026 02:49
@github-actions github-actions Bot removed the optimizer Optimizer rules label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TopK aggregation duplicates NaN groups and can panic during eviction

2 participants