Repository navigation
fix: close the final audit findings and add the database mirror - #449
Merged
Merged
Conversation
A dependent predicate write, a TRUNCATE and a CRDT document delete settle the balance deltas of a cross-shard materialized sum from reads of the source rows. Carry those reads into the Calvin transaction as versioned reads so a concurrent change to a source row aborts and retries the settlement instead of committing a stale delta. A TRUNCATE now joins the other tasks of its request in one transaction, and a zero-net settlement that ships no task still builds a single-vShard class that holds its reads. A gateway forward applies tasks one by one and checks no read, so a request that holds settlement reads commits through the deterministic scheduler. Add recon::before_submit and stmt::before_route fail gates and wire and cluster tests that change a source row between the settlement read and the submit.
Extract the dependent-recon gate into its own module and apply it consistently across pgwire, HTTP and native paths, including multi-statement requests, mixed-verb requests and implicit TRUNCATE transactions. Materialized-sum settlements read their source block in the same request, and balances over several columns settle together. Document that best_effort_non_atomic commits a predicate write and the shipped materialized-sum balance as separate groups. Add wire tests covering the new request shapes.
A multi-slice Calvin write answered with the first participant's affected count, so a coordinator hosting one slice reported only that slice. Each participant's completion ack now carries its slice's plain answer, and the coordinator sums them. A plain count also joins a timeseries slice's install counts. - Add affected_payload to add two write answers in a fixed key order, so every node derives the same result. - Fail the statement when a participant's report does not decode, rather than answer a smaller count. - Fold the summed total once per batch in the response shape, with later count-bearing tasks folding their verb alone. - Refuse a request with several RETURNING tasks before it commits, since one transaction answers one set of rows. - Add wire and cluster tests for per-request counts across hosts.
A mirror takes its rows and schema from its source cluster. Until now only some entry points refused other writers. Add a mirror fence, the set of databases that are not promoted mirrors, kept in step with the catalog by descriptor post-apply and rebuilt on boot and metadata image install. Every write gate consults it: the write funnel, the leader proposal gate, the Calvin submit, the authorized dispatch and CONVERT paths, and the metadata proposer for object DDL. Writes from the mirror applier pass. Grants, quotas, DROP/ALTER/PROMOTE DATABASE and consumer offsets stay allowed. Reads follow the session read consistency. A strong read on a mirror returns STALE_READ_NOT_LEADER on every protocol. pgwire and native sessions select eventual or bounded_staleness through default_read_consistency. HTTP and WebSocket requests carry a read_consistency field. Delta sync, topic publish, ILP, RESP pub/sub and CRDT routes report the refusal with its own error class.
A hard revoke used to end its sessions only when the audit bus consumer handled the event, so a lagging consumer or a dropped broadcast event left revoked sessions open. The credential store now kills the user's sessions itself, before it publishes, for create, update, drop, replicated installs and catalog reloads. A catalog reload also bumps versions and publishes events for users it drops or changes. Tenant moves kill sessions by tenant id through the registry and record an audit row, instead of publishing the tenant id as a user id on the user-keyed bus. The bus consumer now only writes audit rows. It batches ready events into one WAL sync on a blocking thread, records how many rows a lag lost, and files a diagnostics report for a lag or an unfinished batch task. The duplicate top-level session_registry module is removed.
Sessions now hold a binding to the session registry for their whole life. The binding carries the kill signal each protocol loop races against its next read, and it rebuilds a session-lifetime identity at request boundaries when the user, its grants or its API key change. Kill paths cover every protocol (pgwire, native, HTTP streams, RESP, ILP, sync): - API key revoke ends the sessions the key opened, on every node. - MOVE TENANT cutover ends the tenant's sessions in post-apply, and a metadata image that carries a cutover does the same on reload. - BLACKLIST IP accepts WITH KILL SESSIONS, and user kills match the credential-store principal so an OIDC session with an equal numeric id stays open. - Database grant changes bump the user's identity version. MOVE TENANT drain now returns holds that end the drain when the statement future is dropped, so a killed connection strands no drain. The session cap applies per node; the config doc says so.
…al floor Group disk writer: - Retry I/O errors with a doubling wait, reopening the log on each attempt, so a full disk resumes once freed. - Fail-stop on an error no retry clears: the replica leads nothing and answers no Raft request, and the failure is recorded and reported on the health and metrics surfaces. The node keeps serving other groups. - Publish durable progress only for commits that succeeded. Ordinal floor and write gate: - Ask each voter independently during a floor round so a slow or returning voter does not hold the others back. - Record the leadership an admission ran for and recheck it under the lock that appends the entry. On a mismatch the entry passes the gate again, so no leader appends an entry its gate did not admit at its current term. Calvin completion verdicts that conflict with the stored one keep the first verdict, reach a diagnostics capture, and notify no scheduler.
Add begin_sync/finish_sync so a caller can run the fsync with no lock on the WAL. An append during the fsync is not claimed as durable and syncs next. A segment roll between the steps stays safe, and a failed fsync latches the active writer even when a roll replaced the file. Add SegmentScanner, a record reader for a WAL directory that holds no borrow of the WAL, so appends run during a scan. The directory replay functions move into their own module and re-export from segmented.
Record a statement's intent in the audit log before its mutation commits, across DDL, sync, HTTP and native entry points, so a record that fails refuses the action. Rework the audit pipeline around the WAL: - Run the audit WAL fsync with no WAL mutex held, so concurrent recorders share a sync and an append never waits for one. - Bound the in-memory buffer. At the hard bound a recorder with an error path refuses, and a best-effort entry stays in the audit WAL until a flush commits it. - Release recorder waiters only after a flush commits to the catalog. A failing catalog throttles recorders instead of dropping entries. - Run the flush executor on its own blocking thread, and hand off the Tokio worker before an async recorder waits. - Verify the hash chain one entry at a time without holding the log. - Report a batch as durable, staged, or unrecorded to its sender. Bound the audit rows of repeated bearer refusals per peer, and emit periodic summary rows for the throttled count. Report OTLP auth errors through a dedicated type. Stop the JWKS refresh loop at drain, and reject a credential key that reuses the WAL or backup key. Expose audit counters in the system metrics.
Add two invariant reports. One fires when the quarantine registry drops a record at its bound. The other fires when a release of idle descriptor leases is not proposed.
Charge corrupt time-series partition reads to a quarantine record on every scan path, and read partitions on the owning core through its io_uring instead of a scan thread. Release the records of a collection, tenant or partition when its segments are removed, replaced or merged, so a reused segment id starts clean. Bound the registry and report evictions. Read partition cells with a typed error rather than a panic.
An idle group resends its snapshot at the same index, and a replica that awaits a snapshot neither replicates nor campaigns. Refusing a chunk that decodes held the replica behind until a write or a restart. A chunk that decodes now clears its record, while bad bytes stay refused with a stable error that names the quarantine. Installing a snapshot releases the records at or below its index.
A drain has no expiry, so a lost drain end leaves its descriptor drained until a restart. A full queue now hands the end to a backlog that needs no queue slot. An end whose attempts fail files a report and retries until it applies, and shutdown gives each waiting end one bounded attempt. Dropped releases of other kinds file a report too.
A catalog file that exists and does not open, or whose rows do not load, used to seed every core empty and look like a first boot. Each seed loader now returns a typed error that names the file, the cause and the next action, and only an absent catalog file seeds empty. A vector index rebuild that cannot read the store or insert a row fails boot instead of publishing an incomplete index.
…register A register refused after it applied its policy or installed its config now puts back what it replaced. A failed peer-id change keeps the previous incarnation.
…olds it Requeue delivers at least once. The entry stays in the DLQ, claimed and safe from eviction, until the action consumer durably holds it, and an inbox that refuses the action gives the claim back. A requeue by another tenant reports the entry as absent, and requeue writes a durable audit record first.
Extend the durable ActionIntent record to statements whose first effect precedes their catalog entry (row copy, tenant move, backup, engine index change, purge), the JWT key exchange, the session tenant override, a scheduled backup, a restore of a hard-deleted collection, and DEFINE EVENT actions. A failed record refuses the action, and a refused trigger action goes to its retry queue. Actions that remove access stay best-effort so an audit outage cannot block a security response.
Wall-clock reads in nodedb-types and nodedb-array go through web-time, which uses the JS clock on wasm32-unknown-unknown where std::time panics. Add checked_wall_now_ms, which reports a clock before the epoch as None so callers that stamp stored data can refuse instead of storing epoch 0. The fuzz workflow runs the 32-bit build check over the array, graph, FTS, vector and types crates and triggers on the array, graph, FTS, vector and memory crates.
Array column decoders read lengths through shared bounds-checked frame helpers, so an oversized or truncated length is a corruption error and never an out-of-range slice or allocation. The error-details string array decoder reads through the checked Vec reader instead of pre-allocating from the claimed length. MBR building no longer relies on expect and treats an all-null or empty column as AllNull.
ST_BUFFER and GEO_CIRCLE refuse a segment count above the cap with SpatialError::BufferSegments instead of attempting the allocation.
Export the list of window function names from nodedb-query. The planner refuses a registered aggregate or timeseries function that has no window evaluator, and the evaluator returns UnknownFunction instead of panicking.
A row-format COPY reads the file under a memory governor reservation and holds one parsed row at a time, reserving that row too. The first pass checks every row so an unparsable row writes nothing, and the second pass inserts. Raster loads reserve each block from the reader's block_bytes, and an array COPY stream reserves its incomplete trailing record. A refused reservation is SQLSTATE 53200 with the next action.
ArrayPutBatch gets the tenant quota check and max_batch_size cap that edge batches already apply, counting the longest column so a short first column cannot hide a long one. The shared request decoding and size refusal move to one admission module used by both opcodes.
Column redaction does not cover array attributes, so an array read under a policy is refused whole, and CREATE ARRAY refuses a name a policy already holds. CREATE ARRAY stamps its creation time from the checked clock and errors when the clock reads before the epoch.
AS OF NOW(), DEFAULT NOW() and SHOW CHANGES read the checked clock and return a typed error with the next action instead of using epoch 0. A lone quote after AS OF SYSTEM TIME is a parse error, not a panic.
LATERAL top-k and loop joins carry the statement's AS OF to their inner scans and refuse an inner side the by-name scan cannot read: a CTE or derived table, a non-document engine, a valid-time scope, or AS OF SYSTEM TIME NULL. CTE inlining recurses into LATERAL plans, and a search leaf over a CTE or derived table is refused instead of reading a collection of the same name. A statement that names AS OF SYSTEM TIME is refused when no live node can pin its cut. SELECT-list aliases bind in HAVING at any expression depth. Array function dispatch uses typed kinds, and a LATERAL item with no subquery is a typed error.
The tenant QPS series was rendered with a hard-coded "default" database label. Count admitted requests per database and tenant in the database metrics registry, render them as nodedb_tenant_qps_total with the real database name, and remove a dropped tenant's series from every database. Document the per-database series and the tenant-wide series, which carry only the tenant id.
Add a generation counter to the system catalog that advances on every committed write of a collection or array row. The database metrics sampler publishes collection and tenant counts only when that generation or the set of databases changed since the last scan, and keeps nothing cached after a catalog error so the next tick scans again.
Use the scene array's real dimensions and attributes in the ARRAY_AGG and GeoTIFF COPY examples, and drop the TIME option. Show concrete bearer login audit entries, explain the method values, and note that the provider, sub and reason values are quoted and escaped.
The Apache-2.0 license covers the crate's source, not the EPSG data. Name the registry versions and generators behind the shipped data files.
Check that the ring buffer push hook runs once per successful push and never after the consumer drops. Check that a capped graph RAG fusion on the single-node reference emits the MAX_VISITED truncation notice and an uncapped one emits none.
Replace the fixed sleep with a wait on each core logging its checkpoint completion twice in the restarted boot, so the file check runs only after a cycle in which a stale index would have rewritten its files.
Describe what the Calvin determinism tests cover, and point two module docs at the files that exist.
… log Track how far a mirror followed its source as a single MirrorCutState (ready cut, applied cut, cut HLC, system-time cap and per-group resume positions) instead of per-node lag and collection-map rows. A new PutMirrorCutState catalog entry carries it, so every node holds the same row. Apply only raises the row, and writes nothing for a database that follows no source, so a replay or a late entry is a no-op. PROMOTE and DROP remove the row in their apply. Replace the lag record with typed MirrorFailure reasons on the stopped status, and add the source database name to MirrorOrigin. The observer, restart and fence paths, the promote and read handlers, and SHOW DATABASE MIRROR STATUS read the new state. The status output now reports failure, byte progress, lag, ready cut and applied cut. Tests and docs follow the new shape.
Compaction ceilings are now named per owner. The log never compacts past the lowest ceiling, so one owner releasing entries no longer frees entries another owner still needs. Ceilings set before a group mounts bind it when it mounts. Each log tracks the entry bytes it holds in memory. The count shows in group status and in the cluster raft group listing.
A database id is now (space << 30) | counter, where the space derives from the cluster id. Two clusters issue disjoint ids, and every id stays below 2^53 so JSON clients read it exactly. The id space travels in the DatabaseIdReserve metadata entry, so every node computes the same id from the log alone. start_raft installs the space from the saved cluster id before any listener opens. The registry keeps the high-water mark rising when it already passed the entry's space. The native placement test now creates databases until one has a collection name that splits placement, since the id no longer fixes that.
…them A mirror's source cut now travels as a MirrorCut metadata entry. The data entries of a cut commit in every mirror group and wait in their lanes. The cut entry installs the cut's schema and raises the ready cut at one log index, which releases them. An entry from a former applier's epoch, a promoted mirror or a dropped database is a no-op on every node. A cut that equals the ready cut installs nothing and only raises the epoch fence and the applied fields, so a new applier releases a partial cut a former one left in the lanes. Redo records and chunks, the Calvin applied ledger, snapshot capture and install, the write gates and SHOW output carry the cut state to match.
…ompaction A source now answers mirror links. A listener accepts a mirror's connection, checks the client against an allowlist, and tails the committed entries of the source's Raft groups over a bounded read. The mirror side gains a client for the same wire. Each link holds a replicated pin that records the last acked index per group. The pin bounds log compaction, and it expires by age or bytes, or ends when the source database drops. Pin rows replicate through the metadata log so every node holds the same ceiling. A source also takes mirror cuts. A cut carries a mark (number, newest system-time ordinal, covered metadata index) in its Calvin marker and in every group's barrier, plus the fully applied Calvin epoch of each vShard, so a mirror installs no redo that its ledger already holds. Tuning and server config gain the mirror settings, and the old source handler gives way to the new source service.
A mirror cluster now replays a source's tail into its own Raft groups. The applier fences its leadership, cuts each source group's tail into batches, rewrites entries for the mirror's id space, proposes them with a commit-wait propose, and applies each source cut through the metadata log as a MirrorCut entry that carries the frozen catalog entries. Supporting changes: - Add a log probe RPC so a node can compare entries it proposed with what a replica committed, backed by MultiRaft::log_probe_view. - Expose the link id on CrossClusterLink and install the Raft operations the appliers use on MirrorState during Raft start. - Conclude a mirror cut's barrier as durable so an idle group's floor crosses it and compaction runs. - Run compaction on every hosted group from the pin pass, so a log released by an expired pin is freed without a following write. - Cover the applier with a two-cluster test, extend the source tests for idle-group compaction, and raise that test's slow timeout.
Outside the default database the edge store and the CSR hold a collection under its database-qualified name, while the IN clause names it bare. The collection filter now resolves the stored name, and a scoped query with no stored name matches nothing.
An array outside the default database was named bare in its write note, its row-change resolution and its change events, while every other collection name there carries the database qualifier. A transaction writing such an array fail-stopped the core after the redo record installed, and its change events did not resolve on publish. Array writes now name the array qualified in the written-collection list, the CDC metadata and the transaction row changes. A settled redo record that still names a collection outside its database no longer halts the core: the note drops the newest-ordinal cache, applies the LSN and reports the misnamed record.
…ata group PURGE TENANT and CONVERT COLLECTION ran as one Data Plane op on the node that received the statement. They now propose a TenantPurge entry to each data group's log and a CollectionConvert entry to the collection's group, so every replica applies them at one log position. Both are exclusive in the apply loop, with a WAL barrier around the core work, and neither adds storage, so a purge passes at any tenant size. A conversion re-encodes rows from inputs shared by every replica. The catalog row records the conversion (generation, incarnation, source mode), and each core keeps a per-collection conversion record in the sparse store. A core resolves the storage mode its rows actually hold at register and boot, so a replica whose group has not yet applied the conversion keeps reading in the source mode. Tenant snapshots, snapshot install and restore carry or drop the records. The tenant purge also clears graph edges, node labels and chain heads through one group-scoped path.
A new mirror database now fills itself from its source before it follows the source's log. The source captures the database at one cut, including a catalog image and each tenant's conversion records, and streams it in chunks. The mirror installs the catalog image, re-issues the rows under their source keys, surrogates and stamps, and takes its first cut. A mirror status of Bootstrapping admits only the applier's unstamped follower writes, and every other source stays refused. MIRROR DATABASE now contacts the source first. The source's hello names the id the mirror keeps, and the statement refuses a source the config does not name, this cluster as its own source, the source's default database, an unmaterialized clone, a second mirror of one source database, and an id already in use. A per-database applier supervisor on each node starts following once the descriptor exists. A capture that cannot install, or stops part way, fails the bootstrap with a typed error.
A mirror's status now changes through a StepMirrorStatus catalog entry, so every node reaches one answer when the lag monitor, a seal, and a promote race. Each step names the status it read and changes nothing when the row no longer permits it. Only the metadata leader proposes lag steps. A read-only mirror now limits its own history and background work: - version cleanup keeps each mirror's stores at or below its applied cut - KV expiry, continuous aggregate retention, and constraint reconciliation skip mirrored databases, whose changes arrive through the log - reads on a mirror are capped at its applied cut A tenant purge also clears collections a core's stores hold with no catalog row, and removes array cells while keeping the array definition.
A mirror now receives every change its source makes on its own clock, and an operator can recover a mirror whose link stopped. Retention: the metadata leader reads its clock once and proposes each retention decision (timeseries partition drops, bitemporal and audit purges) into every data group's log. Replicas apply the decision at one log position and a mirror's tail carries it, so a mirror reads no clock. Local retention loops skip mirrored databases. Sequences: a node reserves a run of values through a catalog entry that names the row it read, so every node and every promoted mirror counts from one durable counter. GAP_FREE is refused, since a rollback or a node stop leaves a gap. Continuous aggregates: a bootstrap capture carries each root's aggregate state and the mirror installs it after the root's rows. Recovery: ALTER DATABASE ... MIRROR RESUME retries a link from its replicated positions, and MIRROR REBOOTSTRAP starts over. A mirror status gains progress, installed, fail, resume and rebootstrap steps, and a capture records the source's metadata timeline so a restored source fails the link. MIRROR DATABASE claims its name, id and source through one catalog entry, so concurrent statements resolve in log order.
A database mirrored with MODE = sync now holds each committed write until the mirror's groups commit a cut that contains it. A write that gets no confirmation within the new sync_timeout_ms tuning returns MIRROR_SYNC_TIMEOUT (SQLSTATE 55M01). The write is committed on the source, so a client must not retry it as failed. The wait covers SQL DML, graph, KV, CRDT, array, COPY, CALL, native, RESP, HTTP CRDT apply and Prometheus remote write paths, and a transaction block waits at its COMMIT. ILP, CRDT sync sessions and event-plane writes do not wait. CRDT: the Loro root of a collection is now its bare name in every database, so a client delta applies in a named database. HTTP crdt_apply admits the database-qualified collection, and QualifiedCollection gains bare_of_stored. Supporting changes: - the mirror types module is split into a directory of submodules - cluster error conversions move out of error_from into their own file - write admission entry scoping moves to its own module - mirror bootstrap cluster tests are split into focused files - mirror docs describe config, status, recovery, promote and sync mode, and the retention and array docs are updated to match
A DDL statement that returns rows can attach a notice, but the pgwire encoder dropped it. Take each notice out of the results and queue it on the session so the client receives it as a NoticeResponse with the reply.
Reads refused with 55P03 and writes aborted with 40001 leave nothing applied, so the materialized-sum and write-admission cases now run them again instead of panicking or silently skipping the round. Add try_simple_query and error_sqlstate to the raw pgwire harness so a case can inspect an ErrorResponse and keep using the connection. The write-admission round panics on any error other than the two retryable SQLSTATEs.
Drop the entries for files that no longer use the system door and allow the replicated collection-convert applier, which runs a committed entry that carries no user identity.
Handle statements that end right after a backslash escape, after a bare INSERT or UPSERT verb, or after multi-byte text before a vector operator. Each case sliced past the end of the input or off a character boundary. Find the INTO keyword after the verb with any whitespace and leading comments, and return a parse error for UPSERT without INTO. Take the left operand of a vector operator only from a directly adjacent text segment.
…dule Session binding in the security module depended on the server's session_auth module for identity construction. Move the stored-user and API-key identity builders, and session identity rebuild, into control::security::session_identity and point callers at it.
Drop -runs=0 so the ASan job fuzzes for its time budget instead of only replaying the existing corpus.
Refresh Cargo.lock with newer compatible releases of the AWS SDK and smithy crates and related transitive dependencies. Drops the unused pin-utils and num-integer entries.
farhan-syah
added
the
run-ci
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the findings of the final audit of
v0.6.2..main, and adds the working database mirror.This PR also holds the six fixes of #426, which did not reach
main.What changes for a user
Database mirrors follow a source cluster. A mirror keeps a read-only, ordered copy of one database from another cluster.
MIRROR DATABASE <name> FROM <cluster>.<db> [MODE = sync|async]starts a mirror.ALTER DATABASE <name> PROMOTEmakes it writable, at one final cut.ALTER DATABASE <name> MIRROR RESUMEandMIRROR REBOOTSTRAPrecover a failed mirror.MODE = syncholds a write until the mirror has it. A timeout answersMIRROR_SYNC_TIMEOUT(55M01), and the write stays committed.[mirror.clients]allowlist.See the mirror section of the databases guide for config, statuses, refusals, and limits.
These bugs are fixed:
MATCH ... INreturned no rowsREFRESH CONTINUOUS AGGREGATEhung, and retention dropped rows before folding themcrdt_applywas refused, and CRDTUNIQUEandNOT NULLnever firedRESTART IDENTITYdid nothingPURGE TENANTandCONVERTchanged one node onlyINSERTpanicked the preprocessor25006ST_Buffer, and decoders took unbounded or hostile sizesThe audit also covered lifecycle, planner, and low-severity items. The commit log lists each one.
Breaking changes
_system.calvin_appliedchanges its key type.GAP_FREE:CREATE SEQUENCE ... GAP_FREEanswers0A000. No deployment ever held that promise.PROMOTE: a database that never was a mirror answers0A000.KILL USER SESSIONS: it ends stored users. AddOIDCto end an OIDC principal.Where to focus review
TenantPurge,CollectionConvert, andRetentionDropmust apply the same way on every replica.-runs=0, which stopped each target before any mutation.Known limits
bitemporal=truecollections keep their history, and no DDL sets a limit.REFRESH.nodedb-liteneeds a matching update for the newMetaOpfields and sync frame.