Repository navigation
Graph fixes, batch edge writes, graph metrics and read-path speed-ups - #443
Merged
Merged
Conversation
A submit after an idle period now mints an epoch at once instead of waiting out the rest of the epoch period. A submit within one period of the last tick still waits for the regular tick, so a burst fills a single epoch.
The group disk writer notifies the Raft loop after each durable batch. The loop then counts the new durable entries, dispatches the resulting messages, and applies newly committed entries immediately, instead of waiting for the next tick.
Index lookups now carry ORDER BY sort keys and the IndexedFetch handler orders the matched rows before offset and limit apply, so sorted queries no longer fall back to a full scan. Every equality conjunct is a lookup candidate, so a leading equality on an unindexed column no longer hides an indexed one. Computed SELECT items combined with a sort still scan.
- Graph neighbor reads gathered from every core now take a node edges from its home core only, so a cross-shard edge is returned once. - MATCH ... IN <collection> reads a collection that has no catalog row, since edge writes create it implicitly. A soft-dropped collection stays hidden. - A bare SEARCH ... USING VECTOR as the list of an IN predicate selects the compared column instead of *. - Search hits used as a hash join input resolve their body and primary key before the join reads them. - Correct the working-memory KV example in the agent memory guide.
The poller slept on a timer between passes, so a response waited for the next wake. Producers now run an optional hook after each push, and the dispatcher uses it to signal the poller, which waits on that signal with a timeout and routes a response as soon as it arrives. Pruning closed LSNs returns early when no range sits at or below the floor.
A submit now ticks at once whenever every epoch batch the leader proposed has been applied. Submits that arrive while an epoch is in flight queue and are taken as one epoch after its apply, so a burst batches as in group commit. The scheduler driver registers a wake only when a deferred request or cut hold arose during the pass.
A posting row now holds (term_freq, doc_len, positions), so scoring a term reads its postings and document lengths with one table read. The backend trait gains a postings-with-lengths read whose default falls back to reading lengths separately. A document with no recorded length reads back with its term frequency.
With a residual filter, each core now keeps its passing hits cut to top_k, and the Control Plane applies the same filter again to the merged hits. The window widens by the observed pass rate instead of doubling, and a hit already ranked keeps its body.
Only a statement that can open with COPY, INSERT, or UPSERT is tokenized for the routing checks, so the cost of a plain query no longer grows with its length.
Record each sequenced transaction's participant vShards when its epoch batch applies, and send the verdict signal to those schedulers instead of every locally registered one. A transaction with no seeded participants still reaches all of them.
Each RaftNode publishes its term and term-start index at every role change through a LeaderMark. MultiRaft exposes the marks as a LeaderBoard, and the Calvin scheduler reads leadership from it each pass instead of locking MultiRaft.
A WHERE with equalities on several secondary-indexed fields now drives the lookup from one index and carries the others as index probes. The executor keeps only the ids every index holds before it decodes a row body. A vector search with an equality on an indexed field records it as an index prefilter. The index lookup runs first and its surrogates gate the candidate set, so the search ranks only matching rows instead of widening the window until enough pass a residual filter. Fix index_lookup capping a value at 1000 entries and scanning past the value prefix, which dropped rows of a value held by many rows.
GRAPH RAG FUSION previously returned a single text column holding the JSON result. Add a GraphRag plan kind and a shaper that emits one row per fused hit with node_id, rrf_score, vector_rank, vector_distance and hop_distance columns, in rank order. Run metadata is surfaced as a pgwire NOTICE when the memory budget stopped the walk early or the walk reached nodes without an identity binding. Update the graph docs and the wire and cluster tests to read the new row shape.
The group disk writer applied every staged write in its own redb transaction, so a drained run paid one commit per write. Move the per-write bodies into a shared LogOp applier and add RedbLogStorage::commit_ops, which applies a run of ops in order inside one transaction. The writer now commits each drained run once. The single-write LogStorage methods route through the same applier.
GRAPH INSERT EDGE now accepts a list of edges. A clause the current edge already has starts the next edge, and each edge carries its own PROPERTIES, VALID_TIME and UNTIL. An edge missing FROM, TO or TYPE refuses the whole statement. The statement binds its endpoints in one exchange and writes every edge as one unit, which is one Calvin transaction when the edges home on more than one vShard. BatchEdge carries the edge properties so the batch path writes them. The command tag reports the number of edges written.
A sorted indexed fetch over a strict collection converted every candidate to a full msgpack row, one value per column, including wide ones such as embeddings. The sort reads only the columns its keys name, so candidates now sort as rows holding just those columns, and only the rows the sort keeps convert in full. Residual filter matching on strict rows also decodes only the fields the filters reference. Both paths share new helpers that list the columns an expression or filter set reads, and a partial Binary Tuple decoder that falls back to the full decode for tuples written under an older schema version.
…t windows A filtered vector search decodes only the strict-row columns the residual filter reads for each candidate, so a candidate the filter drops never pays for its other columns. Hits that pass the filter then attach whole bodies. Transactions keep whole bodies for the Control Plane to filter after the staged merge. When the index scans flat (no sealed HNSW segment, no IVF), the first window asks for more candidates per wanted hit, since a flat scan ranks every vector regardless of the window width.
- Sparse filter on a sealed segment: the search scores the filter's members in that segment exactly. It skips the filtered beam search when members^2 <= ef * len * m0. Deleted members are skipped. - Tombstone count: the HNSW index keeps a count of deleted nodes. Every delete and undelete updates it, so a live count reads no node. - First filtered window: it widens to the search beam, so the beam is not ranked twice.
Check that a statement mentions the route keyword before running the tokenizer, whose cost grows with statement length.
Read the plain index through a dedicated equality probe and build the prefilter bitmap from the matching keys, without encoding an id list between the index and the bitmap.
Fan a graph hop out to the cores owning the frontier nodes instead of every core, rank GraphRAG expansions from the distance map alone, and merge read versions by insertion.
Graph edge inserts and deletes now run as one unit. A GRAPH INSERT EDGE or DELETE EDGE statement takes several edges, and the native and remote clients send them as one EdgePutBatch or EdgeDeleteBatch request. The batch passes write admission and authorization as a whole. - Graph node and edge gauges refresh from a per-core count on a background loop and report through system metrics and Prometheus. - Transaction reads record a read tree, so nested reads conflict correctly. Graph reads in a transaction see the transaction's staged edges. - Variable-length named pattern expansion and edge-store queries read the temporal cut they are given. - Index lookups and array surrogate scans carry a system-time cut. A vector prefilter reads the same cut as the search it feeds. - The client trait splits its graph methods into a separate definition and dispatch module. - Docs describe the batch syntax and the graph gauges.
farhan-syah
added
the
run-ci
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #436. Closes #437. Closes #369. Closes #372. Closes #376.
Graph fixes, batch edge writes, graph metrics, and read-path speed-ups for indexed, vector and graph requests.
Fixes
GRAPH INSERT EDGEandGRAPH DELETE EDGEtake several edges in one statement. Clients send them as one batch request.response_polleradaptive backoff is unreachable while responses keep arriving — one Control-Plane worker pins a core forever #376: the response poller wakes when a core pushes a response.Behaviour changes
54000.Breaking for Lite
IndexLookupandSurrogateBitmapScancarry asystem_timefield.MetaOp::RefreshGraphCountneeds a match arm.NodeDbGraphtrait holds the client graph methods.Direction::in_pass_keepsreplaces the per-site direction checks.Speed
One VM, one agent-memory data set, default config. Per-request median in ms. Measured before the rebase onto the snapshot-isolation change.
agent_idagent_idindexed: search A falls from 11.5 to 3.15 ms, search B from 4.6 to 1.5 ms.agent_idindex. The planner reads the wide index first.Tests
--retries 0: stage 1 24202/24202, stage 2 533/533.