Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Allow ordinary scalar predicates to use grouped indexes, where the indexed fields can be queried independently, without treating them as ordered composite keys.
A grouped ES index such as
(embedding, category, price)can answer a predicate oncategoryorpriceindependently. Currently the single-field scalar paths exclude this definition altogether, so a normal filtered scan can silently fall back to reading data without opening the ES index. Metadata-only coverage also cannot establish that every shard accepted the predicate.This change:
GlobalIndexerFactory.supportsGroupedIndexScalarQueries()as an opt-in capability, defaulting tofalse. Only the ES factory opts in; there is no ES type-name special case in the core planner.es_search()SQL API.The additional ESLib module dependencies are test-scoped only.
Tests
Added 23 test invocations:
Local validation on upstream base
b21f36cc56ee6bd11f43b7a2d7f43695445cd446uses JDK 11 with Checkstyle, Spotless and Enforcer enabled. The targeted reactor command is:mvn -B -ntp -Ppaimon-eslib -pl paimon-eslib -am \ -DwildcardSuites=none -DfailIfNoTests=false \ '-Dtest=ESIndex*Test,GlobalIndexQueryTest,GlobalIndexEvaluatorTest,CompositeBTreeTableTest,IndexQuerySplitTest,DataEvolutionBatchScanTest,FullTextSearchBuilderTest,VectorSearchBuilderTest,VectorSearchRowFilterExactnessTest' \ packagepackageis used for a fresh worktree so the codegen-loader embedded resources are prepared before the dependent tests run.Result on upstream
b21f36cc56ee6bd11f43b7a2d7f43695445cd446: 380 tests passed, 0 failures, 0 errors, 0 skipped (38 common + 252 core + 90 ESLib). This is a targeted regression run, not the entire repository test suite.CI status
Python / Native CIfailed. The same job on the exact upstream parent, without this change, reported 67 failed, 19154 passed, 131 skipped. The Python source tree and Python CI workflow are byte-identical between this commit and its parent.paimon-python/conftest.pydoes not accept the newbatchkeyword passed by truncate operations. Other baseline failures include native reader/test-fixture incompatibilities and update-order assertions.paimon-rustmain branch and resolved different Rust commits. The fork Native job ended before a final pytest summary; its early termination is not yet explained. This PR does not change Python/Rust code or suppress failing tests.