Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,11 @@
Analytical service/edge watermark lag is separate. Completed-batch writes
have their own gate and write-pool slot; analytical writes use disjoint tables.
Maintenance must hold both gates for checkpointing.
- Batch format 5, including its physical schema, is the only accepted telemetry
format. Schema changes require a format version change. Ingest copies standard
- Batch format 6, including its physical schema and sidecars, is the only
accepted telemetry format. Schema or sidecar changes require a format version
change. Each logs file has `logs.fidx`, a pure function of the file that
commit, compaction and verification derive with one function; trace-detail
logs are read through it in Go, under the spans' pinned catalog capture. Ingest copies standard
semconv keys into typed dedicated columns, converting values of another type
by fixed rules; a value with no conversion leaves the column NULL and is
counted as a spill. The map keeps every value as sent, and the column is the
Expand Down
2 changes: 1 addition & 1 deletion cmd/pitot/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ func main() {
// observability tools, and the annotation route below) are admitted through
// the interactive lane so they cannot oversubscribe the read pool.
queries := observability.New(q, q, cfg.RetentionDays)
admittedQueries := observability.New(panels.AdmitReader(q), q, cfg.RetentionDays)
admittedQueries := observability.New(panels.AdmitReader(q), panels.AdmitTraceDetail(q), cfg.RetentionDays)
writer.SetAdmission(repository.CompactionDebt, panels.InteractiveRefusals)
go func() {
err := writer.Run(ctx)
Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/architecture.json
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@
"id": "parquet",
"type": "database",
"label": "Parquet batches",
"sublabel": "authoritative, trace.fidx",
"sublabel": "authoritative, trace and log indexes",
"pos": [
40,
600
Expand Down
11 changes: 8 additions & 3 deletions docs/diagrams/persistence.json
Original file line number Diff line number Diff line change
Expand Up @@ -127,14 +127,19 @@
"id": "trace",
"type": "backend",
"label": "Indexed trace read",
"sublabel": "trace.fidx lookup",
"sublabel": "trace.fidx and logs.fidx lookup",
"stage": 3,
"row": 0,
"sources": [
{
"path": "internal/telemetry/trace_index.go",
"line": 203,
"end_line": 235
"line": 215,
"end_line": 238
},
{
"path": "internal/telemetry/log_index.go",
"line": 399,
"end_line": 433
}
]
},
Expand Down
160 changes: 160 additions & 0 deletions docs/proposals/0013-log-trace-lookup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Indexed trace logs

Status: Draft
Date: 2026-11-20

## Summary

Trace detail finds a trace's spans through each batch's trace index, then finds
its logs with `trace_id = ?` over every log file in the window. Logs are sorted
by time bucket, namespace, service and time, so `trace_id` has no locality and
every lookup decodes it for every retained log row in the window. This proposal
gives each batch a log trace index, `logs.fidx`, that maps a trace hash to its
log rows (batch format 6), and reads those rows in Go, as spans already are. A
lookup then costs one probe per batch plus the trace's own rows, independent of
retained logs.

## Motivation

pitot-bench on a Hetzner ccx33 pair, `v2026.11.10-alpha`, after a 4-hour
prefill (about 1.9B rows, a third of them logs, 115 GB of Parquet), with 20
UI reads/s:

- The read cache drained the prefill backlog in 2,604 s and kept up afterwards,
so read-cache lag is not the cause.
- The read gate completed 5.5 of 20 operations/s. Every one of the 250
trace-detail timeouts failed in `query trace parquet logs`. Trace detail over
6h–24h windows averaged 8.9 s on the server and was 68–80% of server read
time in the first mixed phases. Drill-down and annotations queued behind it
for DuckDB read slots (718 `server is busy` rejections, 570 drill-down
timeouts), so the 8.9 s includes queueing as well as the scan.

`internal/observability/trace_logs_bench_test.go` (build tag `readbench`)
reproduces the scan on an idle 18-core laptop, one log per span, random trace
IDs, spread over 20 hours:

| Log rows | Batches | Span lookup | Log lookup | Log lookup, bloom prototype | 20 trace summaries |
|---|---|---|---|---|---|
| 10M | 100 | 3.2 ms | 39.5 ms | 10.8 ms | 30 ms |
| 100M | 1,000 | 25 ms | 327 ms | 62 ms | 160–177 ms |

The log lookup grows linearly with retained log rows. A day at 48k items/s is
about 1.3B log rows: about 4 s idle on this machine without pruning, and about
0.8 s with Parquet bloom filters, which still check every row group.

## Goals

- Trace-detail log lookup cost independent of retained logs; p95 under 300 ms
on an 8-vCPU host under ingest with a day retained.
- Exact results under a total order that refines today's.
- Per-batch trace probes that read one index block, for trace detail and
drill-down summaries alike.

## Design

### Log index

Each batch with a logs file gets `logs.fidx` next to `trace.fidx`:

- Header: magic, entry count, posting count.
- Entries sorted by trace hash, 16 bytes: `hash u64 | offset u32 | count u32`.
- Postings: one `u32` `logs.parquet` row number per log row with a nonempty
`trace_id`, grouped by entry and ascending within it.

The hash is the same 64-bit hash of the lowercase hex trace ID that spans sort
by. Readers still compare the exact trace ID and namespace. Writers reject a
file with more rows than `u32` holds; compaction outputs are capped at 25M
rows.

The index is a pure function of `logs.parquet`: decode `trace_id`, hash each
nonempty value, sort `(hash, row)` pairs, write. Commit, compaction and
verification share that one function. Commit and compaction run it on the file
they just wrote; offline verification rebuilds the index and compares bytes.
Memory is 12 bytes per traced row, at most 300 MB for a 25M-row compaction
output; above a threshold the build sorts in hash-range passes.

Spans keep `trace.fidx` and its format unchanged. Logs keep their schema and
sort order.

### Probes

Loading a batch keeps, for each index, the first hash of every block of 256
entries in memory. A probe binary-searches the fences, then reads one block
with `pread`. Both `trace.fidx` and `logs.fidx` use it, for `Trace`,
`TraceSummaries` and `TraceLogs`. Today a probe opens the file and makes about
17 reads.

### Reading logs

`ParquetStore.TraceLogs` replaces the DuckDB query in
`internal/observability/trace.go`, under the same pinned catalog snapshot as
the trace's spans:

1. Skip batches whose log time bounds miss the window.
2. Probe `logs.fidx`; for each hit, open `logs.parquet` with its offset index
and seek to each run of posted rows, reading only the projected columns.
3. Keep rows whose trace ID matches exactly, whose `log_time` is in the window,
and whose namespace matches when the scope names one or the trace's
namespace is exact, as the span read does.
4. Keep the first `limit` rows by `(log_time, service, span_id, batch, row)`,
a total order that refines today's `ORDER BY time`, and redact bodies as
today.

Parsed Parquet footers and offset indexes of recently read batches are kept in
a bounded cache, dropped on retirement, because large compacted files have
large footers.

These reads keep read admission: they take the interactive read slot that the
DuckDB query held, with the same deadline shedding.

### Format

Batch format 6. Format 5 batches are rejected like any other unsupported
format; servers upgrading start with empty telemetry. The read-cache version
does not change. AGENTS.md, the storage and data-layout docs and diagrams, and
the release-notes script follow.

## Alternatives

- **Parquet bloom filters on logs `trace_id`.** No format change and about 5x
faster at 100M rows, but one check per row group, about 26k for a day of
logs. A bloom filter per batch still scans each positive batch, up to 25M
rows after compaction.
- **Bounding the log lookup to the trace's span extent plus a margin**, as
Grafana's trace-to-logs does. Constant cost, but logs outside the margin
disappear; Grafana documents empty results with its default shift.
- **Postings per row group.** Still needs a row map through compaction, and
each hit decodes a whole group's `trace_id`.
- **Sorting logs by trace hash, or a trace-sorted copy.** Loses the time and
service pruning other log reads rely on, or nearly doubles log storage.
- **One combined index for spans and logs.** Compaction merges each signal
separately and in parallel, so a combined file would join the two merges.
- **Remapping input postings through compaction.** Avoids rescanning the
output, but its row map approaches one run per row when log times interleave
across batches (hundreds of MB at 25M rows), and it is where a silent
misattribution would hide.
- **Memory-mapping the indexes.** No measured need over one `pread` per probe;
adds unmap lifecycle and the Linux mapping limit.

## Gates

1. Oracle test: trace detail equals a full scan ordered by the same total key,
for random data after commit and after compaction across generations,
including hash collisions, empty trace IDs, the same trace ID in two
namespaces, logs-only batches and the limit.
2. Offline verification rejects a missing, duplicated or misattributed posting.
3. Benchmark after compaction: log lookup at 100M log rows is flat against 10M
and under 20 ms, including a hit deep in a row group; probe cost per batch
at 1,000 and 10,000 batches.
4. Ingest A/B: commit latency and ingest throughput unchanged within noise;
compaction time, peak memory and index bytes per log row reported.
5. pitot-bench on Hetzner, 4-hour prefill, `--sustained-rate 48000`, 20 reads/s:
no trace-detail timeouts, and the read gate's trace-detail p95 target met.

## Risks

- Index size for traces with few logs per batch: 16 bytes per entry plus 4 per
row. Gate 4 measures it.
- Seek cost inside large body pages. Gate 3 measures a hit deep in a group.
- Startup time if every entry is validated at load; load reads fences only,
and full checks stay in offline verification.
2 changes: 1 addition & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ rtk bash scripts/with-duckdb.sh go test ./internal/query -tags=readbench \

Without the `readbench` build tag this selection runs no benchmarks. The default
generated fixture is regression data, not capacity evidence. Representative
replay uses `PITOT_PANEL_BENCH_DATA_DIR` and must pass format-5 verification.
replay uses `PITOT_PANEL_BENCH_DATA_DIR` and must pass format-6 verification.
Executor measurements exclude browser paint and sustained external ingest;
record result-cache policy and analytical warmup with their results.

Expand Down
7 changes: 7 additions & 0 deletions internal/engine/duck.go
Original file line number Diff line number Diff line change
Expand Up @@ -2711,6 +2711,13 @@ func (d *Duck) Trace(ctx context.Context, query telemetry.TraceQuery) ([]telemet
return d.repository.Parquet.Trace(ctx, query)
}

// TraceDetail pins one snapshot for a trace's spans and logs together.
func (d *Duck) TraceDetail(ctx context.Context, query telemetry.TraceQuery) (telemetry.TraceDetail, error) {
pin := d.pinParquet()
defer d.unpinParquet(pin)
return d.repository.Parquet.TraceDetail(ctx, query)
}

// TraceSummaries pins one snapshot for the bounded multi-trace root pass.
func (d *Duck) TraceSummaries(ctx context.Context, ids []telemetry.TraceIdentity, start, end int64) ([]telemetry.TraceTotals, error) {
pin := d.pinParquet()
Expand Down
2 changes: 1 addition & 1 deletion internal/engine/rollup_snapshot_window_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ func TestRollupSnapshotWindowUsesSignalBoundsAndIncludesUnknown(t *testing.T) {
for i := range captured {
if captured[i].ID == "unknown" {
// Optional footer statistics may be unavailable. Change only this
// captured descriptor; every underlying format-5 file remains valid.
// captured descriptor; every underlying format-6 file remains valid.
captured[i].SpanTime.Known = false
captured[i].LogTime.Known = false
captured[i].MetricTime.Known = false
Expand Down
2 changes: 1 addition & 1 deletion internal/engine/sql_keys.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
"github.com/labstack/pitot/internal/telemetry/semconv"
)

// Batch format 5 copies the standard keys into typed dedicated columns
// Batch format 6 copies the standard keys into typed dedicated columns
// (proposal 0005), converted at ingest to the key's type; the column is the
// key's read value. Telemetry SQL keeps writing attributes['k'] and
// resource['k'] with a literal key, and rewriteKeys replaces each such
Expand Down
2 changes: 1 addition & 1 deletion internal/engine/timestamp_plan_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (
)

func TestNativeTimestampPredicatesPruneParquetAndPreserveNanoseconds(t *testing.T) {
// Use real format-5 files with three disjoint row groups, and a non-UTC
// Use real format-6 files with three disjoint row groups, and a non-UTC
// process timezone. Nanosecond window parameters must not lose precision.
t.Setenv("TZ", "America/Los_Angeles")
store, err := telemetry.OpenParquetStore(t.TempDir())
Expand Down
2 changes: 1 addition & 1 deletion internal/engine/views.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ CREATE OR REPLACE MACRO attr(attributes, key) AS
// A binding descriptor for panel validation, never the execution source.
const emptyResourceInventory = `SELECT NULL::VARCHAR AS signal,NULL::VARCHAR AS namespace,NULL::VARCHAR AS service,NULL::VARIANT AS resource,NULL::TIMESTAMPTZ_NS AS first_seen,NULL::TIMESTAMPTZ_NS AS last_seen WHERE false`

// CreateParquetViews exposes the fixed format-5 schema. Binding one file
// CreateParquetViews exposes the fixed format-6 schema. Binding one file
// rather than unioning every footer bounds binder memory. Supplying a schema
// MAP suppresses DuckDB 2's VARIANT extraction pushdown, so read the physical
// schema directly; startup rejects mismatched batch schemas before this point.
Expand Down
2 changes: 1 addition & 1 deletion internal/observability/batch_reads_bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import (
)

// TestCompletedReadBenchmark measures actual dashboard methods over identical
// immutable format-5 files. Run with the readbench tag on an otherwise idle
// immutable format-6 files. Run with the readbench tag on an otherwise idle
// machine; the normal test gate does not run this resource-intensive fixture.
func TestCompletedReadBenchmark(t *testing.T) {
cfg := config.Config{DataDir: t.TempDir(), DuckDBMemory: "2GB", DuckDBThreads: 4, DuckDBMaxConns: 4, RollupInterval: 100 * time.Millisecond}
Expand Down
3 changes: 2 additions & 1 deletion internal/observability/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ var (

type DB = queryrows.Queryer

// traceReader reads a trace's spans and logs through the batch indexes.
type traceReader interface {
Trace(context.Context, telemetry.TraceQuery) ([]telemetry.IndexedSpan, telemetry.TraceTotals, error)
TraceDetail(context.Context, telemetry.TraceQuery) (telemetry.TraceDetail, error)
}

type Service struct {
Expand Down
33 changes: 7 additions & 26 deletions internal/observability/service_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -198,11 +198,6 @@ func TestTraceSelectsRecentErrorAndCorrelatesLogs(t *testing.T) {
}}); err != nil {
t.Fatalf("commit trace fixture: %v", err)
}
mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)).
WithArgs("trace-1", start, end, true, "prod", 20).
WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}).
AddRow(start.Add(150*time.Millisecond), "ERROR", "checkout", "payment declined", "trace-1", "root").
AddRow(start.Add(160*time.Millisecond), "ERROR", "payments", `charge failed: token=abc123 {"client_secret":"cs_live_9"}`, "trace-1", "child"))

result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "", "checkout", 20)
if err != nil {
Expand All @@ -223,7 +218,7 @@ func TestTraceSelectsRecentErrorAndCorrelatesLogs(t *testing.T) {
}

func TestTraceLogsUseFullScopeEventTimeAcrossBatches(t *testing.T) {
svc, mock, repository := newMockService(t)
svc, _, repository := newMockService(t)
start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC)
batches := []telemetrystore.Batch{
{ID: "trace-latest", Spans: []telemetry.Span{{Namespace: "prod", TraceID: "trace-order", SpanID: "root", StartUnixNanos: start.UnixNano(), DurationMS: 1}}, Logs: []telemetry.Log{{Namespace: "prod", TraceID: "trace-order", TimeUnixNanos: start.Add(30 * time.Millisecond).UnixNano(), Body: "latest"}}},
Expand All @@ -236,13 +231,6 @@ func TestTraceLogsUseFullScopeEventTimeAcrossBatches(t *testing.T) {
t.Fatal(err)
}
}
mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)).
WithArgs("trace-order", start, start.Add(time.Hour), true, "prod", 10).
WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}).
AddRow(start.Add(10*time.Millisecond), "", "", "earliest", "trace-order", "").
AddRow(start.Add(20*time.Millisecond), "", "", "middle", "trace-order", "").
AddRow(start.Add(30*time.Millisecond), "", "", "latest", "trace-order", "").
AddRow(start.Add(2*time.Minute), "", "", "unrelated later event", "trace-order", ""))
result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: start.Add(time.Hour)}, "trace-order", "", 10)
if err != nil {
t.Fatal(err)
Expand All @@ -259,20 +247,22 @@ func TestTraceUsesIndexedParquet(t *testing.T) {
if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "indexed-trace", Spans: []telemetry.Span{{
Namespace: "prod", TraceID: "parquet-trace", SpanID: "root", ServiceName: "checkout", Name: "pay",
Kind: "SERVER", StartUnixNanos: start.UnixNano(), DurationMS: 25, StatusCode: "STATUS_CODE_ERROR", StatusMsg: "declined",
}}, Logs: []telemetry.Log{{
Namespace: "prod", TraceID: "parquet-trace", SpanID: "root", ServiceName: "checkout", Severity: "ERROR",
TimeUnixNanos: start.Add(time.Millisecond).UnixNano(), Body: "token=secret",
}}}); err != nil {
t.Fatal(err)
}
mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)).
WithArgs("parquet-trace", start, end, true, "prod", 10).
WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}).
AddRow(start.Add(time.Millisecond), "ERROR", "checkout", "token=secret", "parquet-trace", "root"))
result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "parquet-trace", "", 10)
if err != nil {
t.Fatal(err)
}
if len(result.Data.Spans) != 1 || len(result.Data.Logs) != 1 || !result.Data.HasError {
t.Fatalf("indexed Parquet trace detail = %#v", result.Data)
}
if log := result.Data.Logs[0]; log.Body != "token=[REDACTED]" || log.SpanID != "root" || log.Service != "checkout" || !log.Time.Equal(start.Add(time.Millisecond)) {
t.Fatalf("indexed trace log = %#v", log)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatal(err)
}
Expand All @@ -294,9 +284,6 @@ func TestTraceCombinesIndexedSpansAcrossBatches(t *testing.T) {
}}}); err != nil {
t.Fatal(err)
}
mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)).
WithArgs("split-trace", start, end, true, "prod", 10).
WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}))
result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "split-trace", "", 10)
if err != nil {
t.Fatal(err)
Expand All @@ -320,9 +307,6 @@ func TestTraceReadsRecentRootFromParquetIndex(t *testing.T) {
}}}); err != nil {
t.Fatal(err)
}
mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)).
WithArgs("new-trace", start, end, true, "prod", 10).
WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}))
result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "new-trace", "", 10)
if err != nil {
t.Fatal(err)
Expand All @@ -346,9 +330,6 @@ func TestTraceFiltersIndexedSpansByNamespace(t *testing.T) {
}}); err != nil {
t.Fatal(err)
}
mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)).
WithArgs("shared-trace", start, end, true, "prod", 10).
WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}))
result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "shared-trace", "", 10)
if err != nil {
t.Fatal(err)
Expand Down
Loading
Loading