Skip to content

perf(telemetry)!: index trace logs per batch (batch format 6) - #507

Merged
vishr merged 13 commits into
mainfrom
perf/log-trace-index
Nov 20, 2026
Merged

vishr merged 13 commits into
mainfrom
perf/log-trace-index

Conversation

@vishr

@vishr vishr commented Nov 20, 2026

Copy link
Copy Markdown
Member

Trace detail found a trace's logs with trace_id = ? over every log file in the window. Logs sort by time bucket, namespace, service and time, so nothing pruned that filter and every lookup decoded the trace_id of every retained log row. On a Hetzner ccx33 with about 1.9B rows retained, all 250 trace-detail timeouts failed in that query, and drill-down and annotations queued behind it.

Each batch with logs now has logs.fidx, which maps a trace hash to the trace's log rows. It is built from logs.parquet by one function used by commit, compaction and offline verification, which rebuilds the index and compares it. Trace detail reads the posted rows in Go through the Parquet offset index, under the same catalog snapshot as the spans, and keeps read admission. Trace index probes for both signals go through in-memory fences and read one block. Parsed footers of recently read trace files are cached, bounded, and dropped on retirement.

Results are exact. Logs are ordered by (log_time, service, span_id, batch, row), a total order that refines the old ORDER BY time. An oracle test compares against a full scan after commit and after compaction.

Breaking: batch format 6. Format 5 data is rejected, so upgraded servers start with empty telemetry.

Laptop benchmark (internal/observability/trace_logs_bench_test.go, 100M log rows):

Before After After, compacted
Log lookup 327 ms 16.0 ms 1.7 ms
20 trace summaries 160–177 ms 53.9 ms 12.5 ms

Commit latency for a 100k-row batch is unchanged within noise. Proposal: docs/proposals/0013-log-trace-lookup.md.

vishr added 13 commits November 20, 2026 10:11
Every batch with logs gets logs.fidx: entries of trace hash, posting
offset and count, then one u32 logs.parquet row per traced row. The
index is a pure function of the logs file. Commit and compaction build
it from the file they just wrote, in hash-range passes above 8M traced
rows, and offline verification rebuilds it and compares every entry
and posting.
Loading a batch keeps the first hash of every 256-entry block of
trace.fidx and logs.fidx. A probe binary-searches the fences and reads
one block with a single pread, instead of about 17 reads per lookup.
The file is still opened per probe, so a catalog of thousands of
batches holds no index descriptors.
ParquetStore.TraceLogs probes each batch's logs.fidx and reads only
the posted rows, seeking to each run and decoding the projected
columns. Rows keep the exact trace ID, window and namespace rule of
the span read, and the first limit rows by (time, service, span ID,
batch, row), which refines the DuckDB query's time order. Trace detail
reads spans and logs from one pinned catalog capture through
TraceDetail, and the MCP path admits that read as one interactive
read, as it admitted the DuckDB log query.
Trace detail, trace logs and drill-down summaries reopened and
reparsed each span or log file's footer and page indexes per read,
which costs as much as the rows read from a large compacted file. A
cache bounded by 64 MiB of encoded footer and page-index bytes keeps
recently read files parsed and open. Retiring a batch drops its
entries, and a held file closes when its last reader releases it.

At 10M log rows compacted, the warm span read drops from 10.4 to 0.9
ms and the log read from 7.0 to 0.9 ms.
Format 6 adds logs.fidx to every batch with logs; format 5 batches are
rejected like any other unsupported format, so upgrading servers start
with empty telemetry. The Parquet physical schema is unchanged. The
panel benchmark replay rebuilds log indexes for tiled batches, and
AGENTS.md, the storage, data-layout and capabilities docs, and the
diagram sources describe the new sidecar.
Random batches with empty trace IDs, one trace ID in two namespaces,
logs-only batches and forced ties are read through logs.fidx and by a
full scan in the same total order, after commit and after every
compaction pass up to the second generation, across windows,
namespace rules and limits. Every compacted batch then passes offline
verification. A crafted hash collision keeps only the exact trace ID
and fails verification.
The byte bound alone let many small uncompacted files hold hundreds of
descriptors. The cache now also keeps at most 512 files open.
Building logs.fidx through a row reader assembled and converted every
row and added about 105 ms to a 100k-row logs commit. Reading the
trace_id chunk's pages directly brings logs commit latency back to the
pre-index 65 ms.
…ex builds on cancellation

Index sidecar loaders now report short content as a format error and return
other read failures unchanged, so a failing device is never classified as a
corrupt batch. The log index build checks its context before every page, so a
preempted compaction stops promptly; verification passes its context too and
cancellation is never treated as corruption. Closing the Parquet store closes
the footer cache's idle files and leaves held ones to close on release.
The panel benchmark assembled logs.fidx with its own copy of the format.
WriteLogIndex exposes the store's builder for tools that assemble batch
directories, so the benchmark can no longer drift from the format.
Trace reads go through TraceDetail, which pins one snapshot for spans and
logs. The trace benchmark now times that read instead of a path production
never takes.
An exported writer for the bench is unreachable from production, which the
deadcode check rejects. The bench keeps its writer, as it does for trace.fidx.
@vishr
vishr enabled auto-merge November 20, 2026 19:21
@vishr
vishr added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit d6f7c96 Nov 20, 2026
31 checks passed
@vishr
vishr deleted the perf/log-trace-index branch November 20, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant