Skip to content

[core] Support scalar queries on grouped ES indexes - #10771

Open
CrownChu wants to merge 1 commit into
apache:masterfrom
CrownChu:fix/grouped-index-scalar-queries-20261018
Open

CrownChu wants to merge 1 commit into
apache:masterfrom
CrownChu:fix/grouped-index-scalar-queries-20261018

Conversation

@CrownChu

Copy link
Copy Markdown
Contributor

Purpose

Allow ordinary scalar predicates to use grouped indexes, where the indexed fields can be queried independently, without treating them as ordered composite keys.

A grouped ES index such as (embedding, category, price) can answer a predicate on category or price independently. Currently the single-field scalar paths exclude this definition altogether, so a normal filtered scan can silently fall back to reading data without opening the ES index. Metadata-only coverage also cannot establish that every shard accepted the predicate.

This change:

  • Adds GlobalIndexerFactory.supportsGroupedIndexScalarQueries() as an opt-in capability, defaulting to false. Only the ES factory opts in; there is no ES type-name special case in the core planner.
  • Allows eligible grouped definitions in eager and reader-side scalar queries, including search pre-filters.
  • Tracks per-field metadata coverage for eligible definitions and uses evaluated coverage when a reader declines a predicate, preserving uncovered ranges in FULL/DETAIL mode.
  • Makes vector/full-text scalar pre-filters consume evaluated coverage rather than treating declined shards as empty matches.
  • Keeps BTree composite-key planning and vector-primary-column selection unchanged. This does not add an es_search() SQL API.

The additional ESLib module dependencies are test-scoped only.

Tests

Added 23 test invocations:

  • 16 real ES archive/table integration cases: scalar equality/ranges/IN/NULL/AND/OR, FAST/FULL/DETAIL, eager and reader-side execution, serialized splits, partial coverage and different definitions, and unchanged vector-primary selection. Assertions compare against index-disabled reads and verify that the ES archive was actually opened.
  • 2 grouped scalar query cases checking that unsupported shards do not claim evaluated coverage.
  • 5 vector/full-text scalar pre-filter cases checking fallback before top-k ranking.

Local validation on upstream base b21f36cc56ee6bd11f43b7a2d7f43695445cd446 uses JDK 11 with Checkstyle, Spotless and Enforcer enabled. The targeted reactor command is:

mvn -B -ntp -Ppaimon-eslib -pl paimon-eslib -am \
  -DwildcardSuites=none -DfailIfNoTests=false \
  '-Dtest=ESIndex*Test,GlobalIndexQueryTest,GlobalIndexEvaluatorTest,CompositeBTreeTableTest,IndexQuerySplitTest,DataEvolutionBatchScanTest,FullTextSearchBuilderTest,VectorSearchBuilderTest,VectorSearchRowFilterExactnessTest' \
  package

package is used for a fresh worktree so the codegen-loader embedded resources are prepared before the dependent tests run.

Result on upstream b21f36cc56ee6bd11f43b7a2d7f43695445cd446: 380 tests passed, 0 failures, 0 errors, 0 skipped (38 common + 252 core + 90 ESLib). This is a targeted regression run, not the entire repository test suite.

CI status

  • The fork CI run for this commit passed all Java jobs, including ESLib/JDK 11, Core/JDK 8 and 11, Spark, Flink and Java E2E jobs.
  • The full workflow is not green: Python / Native CI failed. The same job on the exact upstream parent, without this change, reported 67 failed, 19154 passed, 131 skipped. The Python source tree and Python CI workflow are byte-identical between this commit and its parent.
  • The 52 failure markers visible in the fork log have the same per-file progress sequences as the parent run. One confirmed baseline issue is that the native-commit tracking wrapper in paimon-python/conftest.py does not accept the new batch keyword passed by truncate operations. Other baseline failures include native reader/test-fixture incompatibilities and update-order assertions.
  • These are historical CI comparisons, not a controlled rerun of the complete Native suite: both jobs build the moving paimon-rust main branch and resolved different Rust commits. The fork Native job ended before a final pytest summary; its early termination is not yet explained. This PR does not change Python/Rust code or suppress failing tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant