Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 3 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,13 +317,13 @@ All resource consumption is observable via Prometheus metrics prefixed `nodedb_`

**Per-Database Metrics** (`nodedb_database_*{database="<name"}`):

- `qps` — Requests per second
- `qps_total` — Cumulative requests admitted
- `memory_bytes` — Total memory reserved
- `storage_bytes` — Cumulative storage on-disk
- `connections` — Open connections
- `bridge_queue_depth` — Pending requests in SPSC bridge virtual queue
- `wal_commit_latency_p99` — 99th percentile WAL fsync latency (milliseconds)
- `maintenance_cpu_seconds` — CPU seconds spent in background tasks (per minute, reset window)
- `wal_commit_latency_p99_us` — 99th percentile WAL commit latency (microseconds)
- `maintenance_lease_seconds_total` — Cumulative wall seconds holding a maintenance lease
- `mirror_lag_ms` — Replication lag to follower replicas (distributed deployments)

**Per-Tenant Metrics** (`nodedb_tenant_*{database="<name>", tenant="<tenant>"}`):
Expand Down
21 changes: 18 additions & 3 deletions docs/databases.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,17 @@ Quota fields:
- `priority_class` — `'critical'`, `'standard'`, `'bulk'` (affects WAL group commit order and SPSC scheduling fairness)
- `maintenance_cpu_pct` — max CPU percentage for background tasks like compaction (default 25%)

### Storage Limit Enforcement

NodeDB checks `max_storage_bytes` at write admission, against the cluster total of the database's stored bytes. The total refreshes once per storage sample period.

- A write that can add bytes fails with SQLSTATE `53100` (HTTP 507) when the total is at or over the limit. INSERT, UPSERT, and UPDATE are refused.
- DELETE, TRUNCATE, DROP, purge, compaction, retention, and internal maintenance writes stay allowed, so an over-limit database can shrink.
- A burst can overshoot the limit by one period's writes, because the total lags the stored bytes.
- Before the first complete total after boot, writes pass.
- A tenant `max_storage_bytes` limit applies the same way to the tenant's total over all databases. Each limit is checked on its own.
- To recover, delete data or raise `max_storage_bytes`. Writes resume after the next sample period.

### Tenant Quotas

Tenants can have their own sub-quotas within a database:
Expand Down Expand Up @@ -355,13 +366,17 @@ KILL SESSION 'sid_abc123';
NodeDB exposes per-database metrics for observability:

```
nodedb_database_qps{database="prod"}
nodedb_database_qps_total{database="prod"}
nodedb_database_errors_total{database="prod"}
nodedb_database_collections_total{database="prod"}
nodedb_database_tenants_total{database="prod"}
nodedb_database_memory_bytes{database="prod"}
nodedb_database_storage_bytes{database="prod"}
nodedb_database_connections{database="prod"}
nodedb_database_bridge_queue_depth{database="prod"}
nodedb_database_mirror_lag_ms{database="prod_eu"} -- if mirrored
nodedb_database_wal_commit_latency_p99{database="prod"}
nodedb_database_maintenance_cpu_seconds{database="prod"}
nodedb_database_wal_commit_latency_p99_us{database="prod"}
nodedb_database_maintenance_lease_seconds_total{database="prod"}
```

Similarly, tenant metrics are labeled by both database and tenant:
Expand Down
3 changes: 2 additions & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -587,8 +587,9 @@ a crashed node's descriptor leases block DDL until they expire.
| `tuning.maintenance.clone_sweep_interval_ms` | `NODEDB_CLONE_SWEEP_INTERVAL_MS` | `30000` |
| `tuning.maintenance.constraint_reconcile_interval_ms` | `NODEDB_CONSTRAINT_RECONCILE_INTERVAL_MS` | `1000` |
| `tuning.maintenance.scope_expiry_interval_secs` | `NODEDB_SCOPE_EXPIRY_INTERVAL_SECS` | `60` |
| `tuning.maintenance.storage_sample_interval_ms` | `NODEDB_STORAGE_SAMPLE_INTERVAL_MS` | `30000` |

All three must be positive. `scope_expiry_interval_secs` has a floor of `10`.
All four must be positive. `scope_expiry_interval_secs` has a floor of `10`.
Below that the sweep costs more than the resolution it buys.

**Calvin scheduler settings** (config file only):
Expand Down
2 changes: 2 additions & 0 deletions docs/security/tenants.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ SHOW TENANT QUOTA FOR acme;
EXPORT USAGE FOR TENANT acme PERIOD '2026-03' FORMAT 'json';
```

`max_storage_bytes` is enforced at write admission against the tenant's cluster total over all databases. The total refreshes every storage sample period, so a burst can overshoot by one period's writes. Writes pass until the first complete total exists after boot. A write that can add bytes then fails with SQLSTATE `53100` once the total reaches the limit. Deletes, TRUNCATE, DROP, and maintenance writes stay allowed. See [Databases](../databases.md#storage-limit-enforcement).

## Creating Users for a Tenant

```sql
Expand Down
26 changes: 26 additions & 0 deletions nodedb-bridge/src/wfq.rs
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,17 @@ impl<T> WeightedFairQueue<T> {
.unwrap_or(0)
}

/// Yield `(database_id, depth)` for every virtual queue holding items.
///
/// Empty, not-yet-reaped queues are skipped. Order is unspecified.
/// Walks the queue map once and allocates nothing.
pub fn nonempty_depths(&self) -> impl Iterator<Item = (u64, usize)> + '_ {
self.queues
.iter()
.filter(|(_, vq)| !vq.items.is_empty())
.map(|(&database_id, vq)| (database_id, vq.items.len()))
}

/// Total items across all virtual queues.
pub fn total_depth(&self) -> usize {
self.total
Expand Down Expand Up @@ -303,6 +314,21 @@ impl<T> WeightedFairQueue<T> {
mod tests {
use super::*;

#[test]
fn nonempty_depths_lists_only_queues_with_items() {
let mut wfq: WeightedFairQueue<u32> = WeightedFairQueue::new(64, 100);
wfq.try_enqueue(1, 10).unwrap();
wfq.try_enqueue(1, 11).unwrap();
wfq.try_enqueue(2, 20).unwrap();

let mut depths: Vec<_> = wfq.nonempty_depths().collect();
depths.sort_unstable();
assert_eq!(depths, vec![(1, 2), (2, 1)]);

while wfq.pop_next().is_some() {}
assert_eq!(wfq.nonempty_depths().count(), 0);
}

#[test]
fn single_db_behaves_like_fifo() {
let mut wfq: WeightedFairQueue<u32> = WeightedFairQueue::new(64, 100);
Expand Down
1 change: 1 addition & 0 deletions nodedb-cluster-tests/tests/common_suite/cases/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ mod single_node_calvin_returning;
mod single_node_calvin_two_phase;
mod sql_ddl_cluster;
mod sql_schedule_lease_gating;
mod storage_totals_cluster;
mod surrogate_raft;
mod surrogate_vshard_transfer;
mod sync_collection_schema;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
// SPDX-License-Identifier: BUSL-1.1
//! Cluster-wide stored-byte totals.
//!
//! Each node samples the bytes of the vShards it leads and asks every peer
//! for its share. Every replica stores the same bytes, and only each group's
//! leader counts them. So every node publishes one copy of the data, and the
//! same total on each node. One node's bytes over every vShard is that copy,
//! because each of the three nodes holds a full replica.

use crate::common;

use std::time::Duration;

use common::cluster_harness::{TestCluster, wait_for_report};

use nodedb::control::metrics::storage::query_local_storage_bytes;
use nodedb_types::DatabaseId;
use nodedb_types::id::VShardId;

/// Sample period every node runs in this test, in milliseconds.
const SAMPLE_INTERVAL_MS: &str = "300";
const SAMPLE_PERIOD: Duration = Duration::from_millis(300);

const COLL: &str = "storage_totals_docs";
const ROWS: usize = 200;
const BODY_BYTES: usize = 1_000;

/// The cluster total of the default database on every node, or why not.
fn totals(cluster: &TestCluster) -> Result<Vec<u64>, String> {
cluster
.nodes
.iter()
.map(|node| {
let store = &node.shared.cluster_storage_totals;
if store.has_total() {
Ok(store.database_bytes(DatabaseId::DEFAULT))
} else {
Err(format!("node {} has no complete total yet", node.node_id))
}
})
.collect()
}

#[tokio::test(flavor = "multi_thread", worker_threads = 6)]
async fn every_node_reports_one_copy_of_the_stored_bytes() {
// Every node reads the sample period at spawn.
unsafe {
std::env::set_var("NODEDB_STORAGE_SAMPLE_INTERVAL_MS", SAMPLE_INTERVAL_MS);
}
let cluster = TestCluster::spawn_three()
.await
.expect("spawn three-node cluster");
cluster
.exec_ddl_on_any_leader(&format!("CREATE COLLECTION {COLL}"))
.await
.expect("CREATE COLLECTION");

// The bytes the default database holds before the rows land.
wait_for_report(
"every node publishes a first complete total",
Duration::from_secs(30),
Duration::from_millis(100),
|| totals(&cluster).map(|_| ()),
)
.await;
let baseline = totals(&cluster)
.expect("complete totals")
.into_iter()
.max()
.unwrap_or(0);

let body = "x".repeat(BODY_BYTES);
for i in 0..ROWS {
cluster.nodes[0]
.exec(&format!(
"INSERT INTO {COLL} (id, body) VALUES ('row-{i:04}', '{body}')"
))
.await
.unwrap_or_else(|e| panic!("insert row {i}: {e}"));
}
cluster
.wait_for_full_apply_convergence(Duration::from_secs(15))
.await;

let payload = (ROWS * BODY_BYTES) as u64;
let check = || -> Result<(), String> {
let totals: Vec<u64> = totals(&cluster)?
.into_iter()
.map(|total| total.saturating_sub(baseline))
.collect();
let (low, high) = (
totals.iter().copied().min().unwrap_or(0),
totals.iter().copied().max().unwrap_or(0),
);
if low < payload {
return Err(format!(
"totals {totals:?} above the baseline hold less than the {payload} bytes written"
));
}
// Rows still settle between rounds. One percent covers that.
if high - low > high / 100 {
return Err(format!("the nodes disagree: {totals:?}"));
}
Ok(())
};
wait_for_report(
"every node publishes the same cluster total",
Duration::from_secs(30),
Duration::from_millis(100),
&check,
)
.await;

// Two more periods: the totals hold steady across rounds.
tokio::time::sleep(SAMPLE_PERIOD * 2).await;
wait_for_report(
"the cluster total holds across rounds",
Duration::from_secs(10),
Duration::from_millis(100),
&check,
)
.await;

// One replica's bytes: node 0 read over every vShard, led or not.
let one_copy: u64 = query_local_storage_bytes(
&cluster.nodes[0].shared,
(0..VShardId::COUNT).collect(),
std::time::Instant::now() + Duration::from_secs(10),
)
.await
.expect("read one replica's bytes")
.into_iter()
.filter(|((_, database, _), _)| *database == DatabaseId::DEFAULT)
.map(|(_, bytes)| bytes)
.sum();
let totals = totals(&cluster).expect("complete totals");
for (node, total) in cluster.nodes.iter().zip(&totals) {
assert!(
total.abs_diff(one_copy) <= one_copy / 100,
"node {} reports {total} bytes, one replica holds {one_copy}: \
the cluster total counts one copy",
node.node_id,
);
}

cluster.shutdown().await;
}
12 changes: 10 additions & 2 deletions nodedb-cluster/src/calvin/types/primitives.rs
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,11 @@ mod tests {
},
expected: BTreeMap::from([(
PassiveReadKeyId::kv(
nodedb_types::QualifiedCollection::from_stored("sessions".to_owned()),
nodedb_types::QualifiedCollection::from_stored(
nodedb_types::DatabaseId::DEFAULT,
"sessions".to_owned(),
)
.unwrap(),
b"abc".to_vec(),
),
Some(b"stored".to_vec()),
Expand All @@ -208,7 +212,11 @@ mod tests {
let keys_only = spec.total_bytes();
spec.expected.insert(
PassiveReadKeyId::kv(
nodedb_types::QualifiedCollection::from_stored("s".to_owned()),
nodedb_types::QualifiedCollection::from_stored(
nodedb_types::DatabaseId::DEFAULT,
"s".to_owned(),
)
.unwrap(),
b"k".to_vec(),
),
Some(vec![0u8; 100]),
Expand Down
1 change: 1 addition & 0 deletions nodedb-cluster/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ pub mod routing_membership;
pub mod rpc_codec;
pub mod shard_split;
pub mod snapshot_board;
pub mod storage_share;
pub mod subsystem;
pub mod swim;
pub mod sync_frame_versioned;
Expand Down
5 changes: 5 additions & 0 deletions nodedb-cluster/src/rpc_codec/discriminants.rs
Original file line number Diff line number Diff line change
Expand Up @@ -186,3 +186,8 @@ pub const VSHARD_ARRAY_SHARD_ELEMENTWISE_REQ: u16 = 90;
pub const VSHARD_ARRAY_SHARD_ELEMENTWISE_RESP: u16 = 91;
pub const VSHARD_ARRAY_SHARD_MAINT_REQ: u16 = 92;
pub const VSHARD_ARRAY_SHARD_MAINT_RESP: u16 = 93;

// VShardMessageType discriminants for storage accounting (u16). These mirror
// `crate::wire::VShardMessageType` repr values.
pub const VSHARD_STORAGE_SHARE_REQ: u16 = 100;
pub const VSHARD_STORAGE_SHARE_RESP: u16 = 101;
Loading
Loading