[VibeSWE]

Description

Add an experimental, opt-in advisory partition budget for newly executing queries. With P_DATAFUSION_QUERY_PARTITION_BUDGET unset, execution retains the existing fixed target and explicit overrides.

When enabled, select min(target_partitions, max(min_query_partitions, budget / active_queries, 1)) from an active-query snapshot. Clone the session state while retaining shared runtime, memory pool, stores, caches, tenant catalogs, UDFs, extensions and optimizer hooks. RAII leases cover collection and streaming workers/results, including errors, responsive timeouts and early drops.

This is an advisory planning policy: existing queries are not resized, and it provides no aggregate admission cap, tenant quota, queuing or fairness guarantee. The explicit target remains the ceiling. The default is unchanged.

This draft is independent of severity clustering and the Flight metrics fix. It contains only the policy/configuration, focused tests, and docs/query-partition-budget.md; no benchmark harnesses or result archives.

Validation

• Extracted production source and tests exactly match the standalone Boat-tested implementation.
• 118 query tests passed (one existing production-fixture case ignored), including default/bounds, shared-state/tenant/UDF/hook identities and lease cleanup.
• Standalone application qualification: 108 finite batches / 468 independently checked queries, two 21-case compatibility comparisons, and mixed visibility/persisted restart checks.
• Formatting, whitespace and exact file-scope checks passed for the extracted branch. No full rebuild was repeated for this source-identical split.

Why this remains a draft

Benefits vary by workload. In the standalone policy campaign, local selective C8 throughput rose from 142.83 to 171.77 completed QPS, while controlled-S3 mixed C8 fell from 1.77 to 1.70. Those are three-block descriptive medians from finite batches against synthetic local/controlled-remote data, not sustained capacity or real-cloud evidence. Keep the setting unset pending representative qualification.

The standalone campaign could not qualify successful Flight delivery because of an inherited metrics panic. Successful Flight was separately exercised in the combined campaign after applying the independent fix. CPU-saturating preemption, enterprise tenant HTTP routing, distributed exchange, hot-tier runtime routing, spilling and memory-admission capacity remain unqualified.

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining lifecycle and shared-state requirements.
• added documentation for configuration, defaults and limitations.

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[VibeSWE]

Follow-up review and qualification (2026-11-09)

Decision: hold as draft. I don't recommend merging on current evidence. Defaults stay unchanged. The code review found no defects: leases are released on every path, shared state is preserved, and the unset path returns the shared context. So no code change was pushed. The open question is the benefit, not correctness.

Run setup. One boat.dev large VM, now erased. Base and budget binaries were built from the same tree, toolchain and lockfile. The server was pinned to four CPUs, so the default target is 16. Data was a 1.5M-row, 101-column fixture on MinIO behind 25 ms of tc netem latency, queried over a 24h window. Each arm ran three rotated blocks; figures are medians. All batches matched independent oracles.

query	C	baseline QPS / p50	budget 16, floor 4	budget 16, floor 8
ERROR by body_message	1	0.95 / 1055 ms	0.94 / 1069	0.95 / 1056
ERROR by body_message	8	6.97 / 1101	6.80 / 1138	7.20 / 1085
count by severity	8	7.88 / 993	7.68 / 1017	8.14 / 967
100 most recent rows, all columns	8	1.15 / 6742	1.36 / 5671 (320 vs 471 MiB read)	1.22 / 6354

At C8, floor 4 cost about 2.5% on selective aggregates, and the block ranges don't overlap. It gained 18% on the heavy SELECT * TopK query, where fewer partitions also read 32% fewer bytes. Floor 8 (half the ceiling) was within noise of the baseline everywhere. So whether the budget helps or hurts depends on query shape, which matches the earlier local and remote results. The fixed per-query listing and metadata cost (about 26 LIST_DELIM calls per 24h query) dominates this remote shape, and partitioning doesn't change it.

What would justify merge: a representative production query mix in which CPU-bound concurrent queries (local, hot-tier or cached data) dominate, showing a repeatable net gain at a documented setting. Otherwise, consider closing. Focused tests pass (4/4, release).

🤖 Generated with Claude Code

[VibeSWE]

Closing: no setting gives a net gain

Follow-up qualification (2026-11-09) found no budget/floor setting that improves throughput without regressing remote queries. All checked responses matched independent oracles, and no correctness defects or lease leaks were found. The problem is benefit, not safety.

Test	Result
Remote selective queries (25 ms netem, C8)	about −2.5% QPS at 16/4; floor 8 neutral
Hot-tier and mixed hot/remote	within noise; 8/1 cut mixed TopK throughput 53% and raised p90 125%
Predeclared sweep, 12 budget/floor pairs, local and remote	local gains of +22–25%, but every candidate failed the remote latency guardrail (closest: 32/12, p90 swung −8.8% to +10.1% per block); 15,440 responses checked
Sustained 610 s open loop	equal throughput, slightly better p50/p99, zero errors, counts drained
CPU oversubscription (16 clients on 4 cores)	short-query p99 73.5 s → 130.2 s (one paired run)
Spilling / memory admission	bounded failures similar to base; spills and recovery work

The budget only helps when every query is CPU-bound on local data. Choosing a safe setting needs per-query knowledge of where the data lives, which the planner doesn't have. Enterprise tenant routing and distributed exchange could not be tested in this checkout.

The branch keeps the follow-up audit doc, regression tests and opt-in metrics (7ab86e58) for reference. For the remote small-read workload, the bigger lever measured here is per-query listing: about 26 LIST_DELIM calls for a 24h query window.

🤖 Generated with Claude Code