Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,18 @@ Monitor admission and compaction together:
`pitot_compaction_retired_batches_total` counts input batches retired by
successful publications. Compare its rate with backlog growth.

Diagnose slow trace drill-downs (`POST /api/queries/exemplars`) by stage:

- `pitot_drilldown_stage_duration_seconds{stage}` times `admission` (the wait
for a read slot, on every exemplar run), `ranking` (the completed-trace query
and its rows) and `summaries` (root lookup for the ranked traces). Result-cache
hits are not observed.
- `pitot_read_cache_pending_spans` counts spans in batches the read cache has
not covered yet. Ranking scans them raw, so a large value with slow `ranking`
points at read-cache lag rather than retained data.
- An exemplar run that takes a second or more logs `slow exemplar request` at
info level with the same stage breakdown in milliseconds.

Run Pitot under a supervisor that restarts it on exit: Docker or Compose with
`restart: unless-stopped` (`--restart unless-stopped`), systemd with
`Restart=on-failure`, or Kubernetes. Docker's `HEALTHCHECK` only marks a
Expand Down
7 changes: 7 additions & 0 deletions docs/proposals/0012-bounded-trace-ranking.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,13 @@ is a union, not a join, so its cost is that of the uncached data only. A window
whose start and end fall in the same minute reads footer-pruned Parquet of the
overlapping files exactly, as the generic drill path already does.

A large uncached tail (250k spans or more) is pruned exactly. In the same
snapshot, the query first ranks the covered minutes' `full` rows and, when
they hold at least 21 traces, keeps only tail spans that score at least the
21st (error, then duration; ties kept). Those rows are real in-window spans,
so every top-21 trace scores at least that much, and a pruned span is the
best span of no top trace. The threshold is a constant in the bound query.

### Exactness

If a trace's best in-window span lies in partition P and the trace is in the
Expand Down
10 changes: 8 additions & 2 deletions internal/engine/batch_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -213,13 +213,15 @@ func (d *Duck) refreshReadCaches(ctx context.Context) (rows int64, backlog bool,
}
}
sort.SliceStable(batches, func(i, j int) bool { return newestBatchEvent(batches[i]) < newestBatchEvent(batches[j]) })
pendingCount := 0
pendingCount, pendingSpans := 0, 0
for _, b := range batches {
if !markers[b.ID] && b.Spans+b.Logs+b.Metrics > 0 {
pendingCount++
pendingSpans += b.Spans
}
}
metrics.ReadCachePendingBatches.Set(float64(pendingCount))
metrics.ReadCachePendingSpans.Set(float64(pendingSpans))
pending := []telemetry.BatchMetadata{}
pendingRows, selected := 0, 0
for i := len(batches) - 1; i >= 0; i-- {
Expand Down Expand Up @@ -392,7 +394,11 @@ func (d *Duck) aggregateSources(ctx context.Context, db snapshotSQL, batches []t
}
// Only statements that rank traces pay for the per-batch exemplar source.
if traces {
sources["trace_exemplars"] = d.traceExemplarSource(batches, markers, w)
source, err := d.traceExemplarSource(ctx, db, batches, markers, w)
if err != nil {
return err
}
sources["trace_exemplars"] = source
}
return nil
}
Expand Down
Loading
Loading