Repository navigation
perf(web): send the Home log volume read on its own so it stops holding the other cards - #2525
Conversation
…ng the other cards The log volume read scans raw logs at 15 minute buckets and takes seconds on a large org. It shared one execute-batch request with the time series and facets, and a batch answers at the speed of its slowest member, so those cards waited on it too.
This comment has been minimized.
This comment has been minimized.
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
🍁 Maple PR previewNote Preview resources were removed when the preview label came off. Final commit |
Maple review🟢 Confidence 10/10 · safe to merge. Contained to one page's command; the new test pins logs-alone and fails if Adds
|
Maple: after this shipped✅ Shipped clean. Nothing this change touches got worse in the hour after it deployed. Deployed to
Maple looks once, an hour after the first deploy that carries the merge. |
Problem
On Home, the time series and facets cards often wait several seconds for the log volume card.
On prd,
FetchOverviewFacets,FetchOverviewLogVolumeandFetchOverviewTimeSeriesstarttogether and also end together. From 11-10 to 11-12 their end time (from the start of
pageload Home) wasp50 ~3.0 s and p90 7.2 to 8.7 s.
The server spans in the same traces show which read is slow (prd, last 24 h, traces that hold a log volume read):
logsTimeseries(log volume)servicesFacetstracesAllMetricsTimeseries(time series)Cause
apps/web/src/page/overview/command.ts:161:FetchLogSeriesgoes through the shared executebatcher, like the time series and facets.
apps/web/src/data/execute-batcher.ts:18: a 15 ms window puts reads from separate commandsinto one
/execute-batchrequest, andpackages/domain/src/query-engine.ts:727says it plainly:the response is all or nothing, so a batch answers at the speed of its slowest member.
logs rollup cannot answer (
canUseLogsAggregatesHourly,packages/query-engine/src/ch/queries/logs.ts:306),so it scans raw logs and reads the
ResourceAttributesmap on every row for the environment filter.On Maple's own org (5.3 M logs a day) that map read alone takes the query from ~0.37 s to ~1.2 s.
Fix
ExecuteBatcherandWarehouseExecutorgetexecuteAlone: one query, sent at once as a batch of one.outsideSharedBatch(effect)(@maple/web-data/warehouse/warehouse-services) runs an effect withits warehouse reads on
executeAlone.FetchLogSeriesuses it. The time series and facets keep batching with each other and stopwaiting for logs. The log volume card itself is not faster (that needs a minutely logs rollup,
see below).
Cost: Home makes one more request per load.
Before / After
pageload Homestart)command.test.ts(real commands and batcher, simulated send)/execute-batchrequests per Home load (3 cold loads, signed in)countalone; traces time series + services facets togetherFollow-up (not in this PR)
The log volume read itself (
logsTimeseriesp95 4.4 to 4.7 s on 11-10 and 11-12) needs a minutely logs rollupwith
DeploymentEnvas a column, spliced likeservice_overview_minutely. That is a newmaterialized view, a migration and a local schema bump, so it gets its own PR.
How to verify after deploy
Home children end time by day (run_sql). Time series and facets should end well before log volume:
logsTimeseriesp95 by day (should not change with this PR; it is the follow-up's target):Checks run
bunx vitest run src/page/overview src/data/execute-batcher.test.tsinapps/web: 56 passed.The new test fails without the
outsideSharedBatchline (one batch holding both reads).bun run --filter=@maple/web typecheck,--filter=@maple/web-data typecheck,oxlint -c .oxlintrc.effect.jsonon the changed files.🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Summary by CodeRabbit