[VibeSWE]

Description

Add experimental opt-in log severity clustering during staging-to-Parquet conversion. P_LOG_SEVERITY_CLUSTER_COLUMN is unset by default; unset or inapplicable streams use the existing writer unchanged.

For applicable log files, buffer one row group's rows and stably reorder them by the configured string/integer column ascending, then the timestamp descending, with nulls last. Preserve row-group membership, schemas, values, timestamp units, row counts, file paths and manifest format. Clear footer ordering claims on clustered files. Metrics, traces, internal streams and missing/unsupported keys retain the existing path.

This draft is independent of partition budgeting and the Flight metrics fix. It includes the writer/configuration, focused writer/crash-recovery/query tests and docs/log-severity-clustering.md; no benchmark harnesses or bulk evidence archives.

Validation

• Extracted production source and tests exactly match the separately Boat-tested clustering branch.
• Full standalone suite: 1,315 library tests plus one integration test passed; final focused clustering/crash-recovery/ordering checks: 75 passed, one existing fixture case ignored. Clippy completed with only pre-existing warnings.
• Independent writer oracles, skip-path byte identity, schema evolution/nulls, partial groups, custom partitions, exactly-once crash recovery and production-scan ordering/filter/aggregate/LIMIT/OFFSET cases passed.
• Actual application three-writer / three-restarted-reader matrix passed, including old readers of clustered files and new readers of old files. HTTP and OTLP/HTTP JSON logs/metrics/traces passed; physical Parquet layout was checked independently.
• Formatting, whitespace and exact file-scope checks passed after extraction. No full rebuild was repeated for this source-identical split.

Why this remains a draft

In the standalone 3.6-million-row MinIO-backed campaign, severity-grouping logical reads fell from 7.24 to 3.68 MiB. Conversion CPU increased 17–19% and peak conversion RSS increased roughly 280–295 MiB; narrow time-only queries read 16–24% more. No reliable local query-latency gain was established. These synthetic results do not justify default enablement or production rollout.

Queries without complete ordering can return rows in a different order, and floating-point aggregation can change in its last bits. Existing files are not rewritten. Kafka, live-tail and OTLP gRPC compatibility, real-cloud latency, sustained ingestion capacity and production-scale memory cost remain unqualified. The standalone campaign encountered the inherited Flight metrics panic; successful Flight was separately qualified in the combined campaign with its independent fix.

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining physical ordering and footer metadata.
• added documentation for configuration, costs and limitations.

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[VibeSWE]

Follow-up review and qualification (2026-11-09)

Decision: hold as draft. Defaults are unchanged. The memory cost is much lower now, but the benefit on this remote shape is still modest and the CPU cost is still unmeasured.

Fix pushed (8c3ffb1). Clustering used to concatenate each row group and then build a full reordered copy, so it held two decoded copies of the group at once. It now orders rows using only the key columns and gathers them from the buffered slices with interleave, 8,192 rows at a time. Already-ordered groups are written as buffered. Row-group membership, order, schema and footer claims are unchanged. Two new tests:

• chunked_writer_matches_single_batch_write_bytes: the Parquet output is byte-identical to writing each reordered group as one batch, which is what the previous implementation did.
• interleaved_batches_match_reference_order: an independent stable sort gives the same order, across uneven slices and output sizes from 1 to 8,192 rows.

Focused release run: 77 passed, 1 existing fixture case ignored (clustering, crash recovery and ordering tests).

Run setup. One boat.dev large VM, now erased. All four binaries used the same source tree, rustc 1.96.0 and Cargo.lock, release profile, and the same stub UI; each was rebuilt incrementally after checkout. The server was pinned to four CPUs against MinIO. The fixture is 1.5M rows with 101 columns (about 1.1% ERROR, 0.5% missing severity) and default 262,144-row groups. Each arm ran three rotated blocks; figures are medians.

Conversion	baseline	first impl (980f350)	current
RSS rise above RSS at ack	0 MiB	810 MiB (724–815)	143 MiB (0–205)
Peak RSS after ack	2,127	2,919	2,424

Process CPU varied by up to 40% between runs of the same binary, and the variation followed run order, so this matrix does not resolve the CPU cost. The earlier +17–19% figure was measured on the first implementation.

Queries. These read the baseline-written and clustered fixtures with the base binary, through 25 ms of tc netem latency to MinIO and a 24h window. All 36 batches matched independent oracles (exact top-100 ERROR groups, severity counts, ordered recent rows, and 1.5M distinct rows).

query	C	unclustered QPS / p50	clustered QPS / p50	logical MiB
ERROR by body_message	1	0.95 / 1055 ms	0.95 / 1069 ms	3.61 → 1.70
ERROR by body_message	8	6.97 / 1101 ms	7.29 / 1056 ms	3.61 → 1.70
count by severity	8	7.88 / 993 ms	8.55 / 913 ms	0.48 → 0.03
100 most recent rows, all columns	8	1.15 / 6742 ms	1.25 / 6274 ms	470.6 → 472.1

Bytes fall sharply, but latency improves by only about 0–8%. At this window, roughly 26 LIST_DELIM calls, a few GETs and 1 PUT per query dominate (a 1h window takes 0.35 s instead of 0.96 s), and clustering doesn't touch that. Stored bytes are 0.3–1.1% larger.

What would justify merge: a clean CPU measurement of the current writer, a narrow time-window query without a severity filter (the earlier run read +16–24% more), and real-cloud or blog-scale evidence. Kafka, live tail, OTLP gRPC and spilling remain unqualified. Raw evidence is kept locally; none of it is in this PR.

🤖 Generated with Claude Code

[VibeSWE]

Follow-up 2: conversion CPU and narrow windows (2026-11-09)

Doc-only update pushed (b0da249); the writer code is unchanged since 8c3ffb1. One boat.dev large VM, now erased. Every query result was checked: 6M distinct rows per fixture, exact 24h ERROR and severity oracles, ordered window rows, and window counts consistent with per-service sums. Zero failures.

Writer CPU. Same 1M-row × 101-column input, production writer settings, each variant in its own process, median of 5:

Writer	CPU	Peak memory rise
Default, 2,048-row staged batches (today)	5.58 s	152 MiB
Default, coalesced to 8,192	4.02 s	244 MiB
Clustered, current (8,192-row interleave)	4.54 s	182 MiB
Clustered, first implementation	3.09 s	483 MiB

Clustering's sort and gather cost about 13% more CPU than a default writer at the same batch size. But the clustered writer hands the encoder 8,192-row batches instead of the staged 2,048-row ones, so against today's path its writer CPU is lower. Whole-server CPU varied too much between runs of the same binary to resolve this at app level.

Side finding, outside this PR: coalescing staged batches before writing would cut default-writer CPU by about 50% (to 2.70 s at 65,536 rows) for about 90 MiB more memory.

Queries at 6M rows, 25 ms netem, C8 QPS:

Query	Unclustered	Clustered	Logical MiB
ERROR groups, 24h	4.02	4.21	19.5 → 11.0
severity count, 24h	4.69	5.08	1.87 → 0.09
recent 100 rows, 1 s window	1.74	1.74	53.9 → 107.1
count, 1 s window	18.84	21.99	0.92 → 1.60
recent 100 rows, 10 s window	1.29	1.52	418.9 → 289.1
ERROR groups, 10 s window (C1 / C8)	2.45 / 16.83	2.16 / 15.60	6.88 → 3.30

A 1 s unfiltered window reads up to 2× the bytes, but no latency regression was measured. The one consistent regression is a severity filter combined with a narrow window: 7–12% lower throughput while reading half the bytes, cause not yet identified.

Remaining before merge: the cause of that regression, and real-cloud or blog-scale evidence. The PR stays a draft with clustering off by default.

🤖 Generated with Claude Code