Speed up weekly forecasts and recent-page loading (1.6.2)

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## Unreleased
 
+## 1.6.2 — 2026-10-19
+
+- Speed up weekly quota forecasts by reading exact selected-week calls and aggregating older EMA samples and calibration history in SQL, preserving the forecast formula and dated subscription prices.
+- Batch Sites metadata and read quota events across active machines with bounded keyset pagination. Measured uncached forecast requests on the private Site fell from 9.7 seconds to 2.1–4.0 seconds; a full reload displayed the chart in 4.4 seconds.
+- Cache completed days from recent ranges, including the last seven days, with atomic preparation and invalidation after late writes. Today and partial days remain live.
+- Return completed reads during concurrent synchronization without caching them, clear stale quota ETags, and time out requests whose response headers never arrive.
+- Reuse bounded request-local pricing summaries without sharing custom prices between requests.
+
+Docker: `capitaine/codex-usage-dashboard:1.6.2` and `:latest` (Linux AMD64 and ARM64). Recreate containers with their existing configuration and storage volumes.
+
 ## 1.6.1 — 2026-10-19
 
 - Read Codex's Zstandard-compressed `.jsonl.zst` session archives directly, alongside plain JSONL logs. Compression no longer makes historical usage disappear locally or propagate as Mesh deletions.

Dockerfile

@@ -20,7 +20,7 @@ LABEL org.opencontainers.image.source="https://github.com/capisoft-lib/codex_usa
       org.opencontainers.image.licenses="AGPL-3.0-or-later" \
       org.opencontainers.image.title="Local Usage Dashboard for Codex" \
       org.opencontainers.image.description="Independent local dashboard for Codex usage, token activity, and cost estimates" \
-      org.opencontainers.image.version="1.6.1"
+      org.opencontainers.image.version="1.6.2"
 
 ENV NODE_ENV=production \
     HOST=0.0.0.0 \

README.md

@@ -6,7 +6,7 @@
 
 Local Usage turns the Codex session metadata stored on your computer into a fast, privacy-conscious usage dashboard. It can run entirely on one machine, or several machines can send signed, minimized usage snapshots to an optional central dashboard hosted with OpenAI Sites or on your own server.
 
-[**Stable release 1.6.1**](https://github.com/capisoft-lib/codex_usage/releases/tag/v1.6.1) · [Docker image 1.6.1](https://hub.docker.com/r/capitaine/codex-usage-dashboard) · [AGPL-3.0-or-later](LICENSE) · [Changelog](CHANGELOG.md) · [CI status](https://github.com/capisoft-lib/codex_usage/actions/workflows/ci.yml)
+[**Stable release 1.6.2**](https://github.com/capisoft-lib/codex_usage/releases/tag/v1.6.2) · [Docker image 1.6.2](https://hub.docker.com/r/capitaine/codex-usage-dashboard) · [AGPL-3.0-or-later](LICENSE) · [Changelog](CHANGELOG.md) · [CI status](https://github.com/capisoft-lib/codex_usage/actions/workflows/ci.yml)
 
 Setup guides: [deploy the central dashboard with OpenAI Sites](docs/sites-deployment.md) · [deploy the public Mesh ingress](docs/mesh-ingress.md) · [install a reporting agent](docs/reporting-agent.md)
 
@@ -165,7 +165,7 @@ The dashboard image runs as a non-root user with all Linux capabilities removed.
 The public Linux AMD64/ARM64 image is:
 
 ```text
-capitaine/codex-usage-dashboard:1.6.1
+capitaine/codex-usage-dashboard:1.6.2
 ```
 
 The 1.0.2 release is also mirrored at `ghcr.io/capisoft-lib/codex-usage-dashboard:1.0.2`.
@@ -174,7 +174,7 @@ On Windows PowerShell:
 
 ```powershell
 $codexData = Join-Path $env:USERPROFILE ".codex"
-$image = "capitaine/codex-usage-dashboard:1.6.1"
+$image = "capitaine/codex-usage-dashboard:1.6.2"
 
 docker pull $image
 docker volume create codex-usage-dashboard-storage
@@ -198,7 +198,7 @@ docker run -d `
 On macOS or Linux:
 
 ```bash
-IMAGE="capitaine/codex-usage-dashboard:1.6.1"
+IMAGE="capitaine/codex-usage-dashboard:1.6.2"
 
 docker pull "$IMAGE"
 docker volume create codex-usage-dashboard-storage

docs/dockerhub-readme.md

@@ -6,7 +6,7 @@
 
 Local Usage turns the Codex session metadata stored on your computer into a fast, privacy-conscious usage dashboard. It can run entirely on one machine, or several machines can send signed, minimized usage snapshots to an optional central dashboard.
 
-[**Stable release 1.6.1**](https://github.com/capisoft-lib/codex_usage/releases/tag/v1.6.1) · [Docker image 1.6.1](https://hub.docker.com/r/capitaine/codex-usage-dashboard/tags) · [Source code](https://github.com/capisoft-lib/codex_usage) · [AGPL-3.0-or-later](https://github.com/capisoft-lib/codex_usage/blob/main/LICENSE) · [Changelog](https://github.com/capisoft-lib/codex_usage/blob/main/CHANGELOG.md) · [CI status](https://github.com/capisoft-lib/codex_usage/actions/workflows/ci.yml)
+[**Stable release 1.6.2**](https://github.com/capisoft-lib/codex_usage/releases/tag/v1.6.2) · [Docker image 1.6.2](https://hub.docker.com/r/capitaine/codex-usage-dashboard/tags) · [Source code](https://github.com/capisoft-lib/codex_usage) · [AGPL-3.0-or-later](https://github.com/capisoft-lib/codex_usage/blob/main/LICENSE) · [Changelog](https://github.com/capisoft-lib/codex_usage/blob/main/CHANGELOG.md) · [CI status](https://github.com/capisoft-lib/codex_usage/actions/workflows/ci.yml)
 
 ## Donate
 
@@ -22,7 +22,13 @@ If you like this project and want to support its ongoing development, you can [b
 - project, model, period, usage, and conversation filters;
 - nine interface languages.
 
-## New in 1.6.1
+## New in 1.6.2
+
+- Faster weekly quota projections with SQL aggregation of historical calibration and EMA samples, preserving the forecast formula.
+- Cached completed days for recent ranges, including the last seven days; current-day activity stays live and late writes invalidate affected days.
+- Completed reads remain available during synchronization, with bounded quota-event pages and batched Sites metadata.
+
+## Included from 1.6.1
 
 Restore historical usage from Codex's compressed `.jsonl.zst` archives without extracting them. Compressed and plain copies keep the same session identity, calls and tokens. An unreadable archive preserves the last successful snapshot instead of triggering historical deletions. The image includes the required Zstandard-capable Node runtime; keep the existing storage volume when upgrading.
 
@@ -37,18 +43,18 @@ Restore historical usage from Codex's compressed `.jsonl.zst` archives without e
 The public image supports Linux AMD64 and ARM64 and runs as a non-root user:
 
 ```text
-capitaine/codex-usage-dashboard:1.6.1
+capitaine/codex-usage-dashboard:1.6.2
 ```
 
 ```bash
-docker pull capitaine/codex-usage-dashboard:1.6.1
+docker pull capitaine/codex-usage-dashboard:1.6.2
 docker compose up -d
 ```
 
 Inspect the published multi-platform manifest and digest:
 
 ```text
-docker buildx imagetools inspect capitaine/codex-usage-dashboard:1.6.1
+docker buildx imagetools inspect capitaine/codex-usage-dashboard:1.6.2
 ```
 
 The published manifests include SBOM and provenance attestations. See the [full README](https://github.com/capisoft-lib/codex_usage#readme) for scoped read-only mounts, PowerShell/macOS/Linux commands, Compose configuration, and hardened deployment examples.

docs/page-loading-performance.md

@@ -0,0 +1,68 @@
+# Page loading measurements — 2026-10-19
+
+Work branch: `codex/page-loading-performance`; release branch: `codex/page-loading-release`. The initial dirty working tree was preserved. Existing local changes were included in the baseline and excluded from the release PR. The final forecast changes were deployed to private Site version 54; hosted measurements appear below.
+
+## Hosted baseline
+
+Sites Worker logs captured before the changes contained 45 completed quota-detail requests returning HTTP 409 between 04:15 and 05:19 UTC on October 19. Median server wall time: **8,431 ms**, range **7,670–13,175 ms**. One additional incomplete quota request was excluded. This is server time, not browser rendering time.
+
+The quota endpoint discarded completed reads when ingestion advanced its revision. Frequent synchronization could therefore keep the graph unavailable. The revised endpoint returns the completed forecast without an ETag, with `private, no-store` and no exact revision marker. Regression tests cover this case and stale ETag removal.
+
+## Same-data local comparison
+
+Node 24.19.0, SQLite, three sequential samples per case. A consistent backup of a local database was frozen before comparing both implementations: **10,122 sessions and 226,520 calls**. It was read-only during both runs. The before implementation is the source preserved before optimization, including prior local edits. SQL time, database round trips and adapter bytes are recorded by the benchmark. Metadata/authentication, network latency and browser rendering are outside this measurement. These cases disable durable rollups to isolate the read-path changes.
+
+| Scenario | Before, median | After, median | Observation |
+| --- | ---: | ---: | --- |
+| Overview, all history | 2,341 ms | 2,295 ms | About 2%; small enough to treat cautiously |
+| Projects, all history | 2,101 ms | 2,102 ms | Essentially unchanged |
+| Conversations, all history | 2,296 ms | 2,083 ms | About 9% faster |
+| Overview, current week | 74 ms | 69 ms | About 7% faster without rollups |
+| Quota detail with calibration history | **10,098 ms** | **1,326 ms** | **7.6× faster** |
+
+Quota database round trips decreased from **78 to 30**, and adapter output decreased from **79.6 MB to 50.4 MB**. Full serialized page/forecast hashes matched before and after for all five scenarios. No counts, ordering or monetary outputs changed in this fixture.
+
+## Last-seven-days cache
+
+A separate deterministic fixture contains 4,096 sessions and 262,144 calls, spread over 70 days, including the current month. Five samples compare the same seven-calendar-day overview before and after warming the available cache:
+
+| Scenario | Before | After |
+| --- | ---: | ---: |
+| Recent-week overview, median | **136.70 ms** | **55.03 ms** |
+| Initial preparation, outside the timed warm reads | 0.70 ms | 48.23 ms |
+
+The warm overview is **2.5× faster** in this synthetic measurement. Serialized results matched exactly. This is a local cache benchmark, not an observed Sites production gain.
+
+Only completed UTC days are reused. Today and partial/split days stay live. Late inserts, corrections and deletion invalidate affected days through the existing triggers. Preparation targets the requested range, prioritizes recent days and writes up to 128 session-days in four atomic statements. Custom prices and project grouping remain request-specific. Raw calls for quota forecasts remain exact; daily aggregates do not replace their timestamps.
+
+## Reproduce
+
+Use Node 22.15+ (measurements above used Node 24.19):
+
+```powershell
+node scripts/benchmark-pages.mjs
+# Warm recent-week cache on an isolated synthetic in-memory database:
+$env:BENCH_ROLLUPS = '1'
+$env:BENCH_SCENARIOS = 'overview-week'
+node scripts/benchmark-pages.mjs
+```
+
+`BENCH_SOURCE_ROOT` selects an alternative preserved source tree. `BENCH_DATABASE` selects an existing SQLite file, always opened read-only; omit it for the generated fixture. `BENCH_PASSES` changes the sample count. Frozen data and raw measurement JSON are retained under ignored `.cache/page-performance/`; no user history is committed.
+
+Functional validation covers owner/revocation isolation, events spanning quota batch boundaries, invalid dates, exact forecast output, cache invalidation, current-day freshness, graph/range boundaries, concurrent ingestion, request timeout and cancellation. At this initial checkpoint the root build/check passed 355 tests; the Sites build and 15 tests passed, including real workerd/D1 execution; TypeScript passed. Deployment and browser measurements followed in the forecast work below. These counts include the pre-existing local features; the isolated release is validated separately.
+
+## Weekly forecast follow-up
+
+The owner reported the projection chart still loading slowly after Site version 52. An authenticated in-browser request to its quota-detail endpoint took **9,735 ms**, returning 2,720 selected-week calls and about 1.11 MB. Worker logs showed 9–12-second successful reads: the older calibration still retrieved several months of individual events.
+
+The revised path retains individual events for the selected week. Older EMA history is grouped by its exact hour relative to the forecast anchor, UTC pricing day, model, tier and validity. Calibration is grouped separately across all comparable quota windows, keeping inclusive observation boundaries. The forecast's 28-day lookback, historical calibration, invalid gaps and pricing rules remain unchanged. No price totals are persisted. `Server-Timing` separates metadata, current calls, historical aggregation, revision validation and forecast calculation.
+
+Using the same fixed local database and the captured 44-period metadata, three samples reduced adapter results from 167,769 rows / 37.44 MB / 23 reads to 3,472 rows / 0.83 MB / 5 reads. Local medians were 1,100 ms before and 734 ms after; this local CPU/database comparison excludes Sites network round trips. Selected-week calls matched exactly and complete forecasts matched within floating-point summation tolerance. Six additional tests cover SQL aggregation, inclusive boundaries, plan differences, invalid usage/dates, gaps and historical selection. The root suite now passes 361 tests; Sites' 15 tests and TypeScript also pass.
+
+A browser measurement of the version-52 refresh completed the forecast chart in **9,960 ms**, with **9,571 ms** in the detail request and approximately **53 ms** between its response and DOM update. Version 53 reduced the detail request to **3,904 ms**. Server timings identified remaining sequential per-machine reads and two sets of three metadata requests. The next revision batches metadata atomically and reads selected-week events across active machines with one shared keyset cursor. Tests verify cross-machine paging and both metadata adapter paths.
+
+Version 54 was deployed successfully to the same owner-private Site. Three authenticated, uncached detail requests measured **3,971 / 2,767 / 2,112 ms** (all HTTP 200, ready forecasts). A full page reload displayed the forecast SVG after **4,408 ms**, including startup and metadata; its detail request took **2,148 ms**. These are observed production/browser timings, not local estimates. Evidence, Server-Timing headers and a screenshot are retained in `.cache/page-performance/weekly-chart-final.json`, `weekly-graph-reload-final.json` and `weekly-forecast-deployed.png`.
+
+## Isolated 1.6.2 release validation
+
+The release excludes the pre-existing project allocation and grouping edits. Its root check passes **354 tests**; the Sites build and **15 tests**, TypeScript and lint pass. A Docker build was exercised with a read-only root filesystem, isolated test volume, relational migration, replacement-container persistence, local collection and compressed-archive recovery. No existing user volume or running application container was replaced by these tests. A request-local summary regression verifies repeated groups, count preservation and custom-price isolation.

docs/relational-storage.md

@@ -58,12 +58,16 @@ After data compaction, rolling back to an old application requires restoring a m
 
 ## Historical aggregates
 
-Migration `0008` adds a durable daily aggregate cache. Summary reads covering at least 28 days use it for complete UTC days in finished UTC months. Calls remain available for details, pricing exports and short zooms. Any day touched by a graph boundary or a partial range is read from detailed rows, preserving local-time month boundaries and inclusive endpoints.
+Migration `0008` adds a durable daily aggregate cache. Summary reads covering at least one day use it for complete UTC days before today, including the last seven days in the current month. Today's activity, partial range edges and days split by graph boundaries remain on detailed rows. Boundaries exactly at UTC midnight allow cached days. Calls remain available for details, pricing exports and short zooms, preserving local-time boundaries and inclusive endpoints.
 
-The cache retains session identity, model, effort, tier, pricing date, context bands, cache evidence, token sums and event counts. Prices remain dynamic. Each request prepares up to 25 pending days in one atomic batch; days not yet prepared use the existing detailed query, so the first visits can still be slower. Automatic page polling continues preparation even when an unchanged response returns HTTP 304. Subsequent reads reuse the durable cache. There is no global history lock or partial result.
+The cache retains session identity, model, effort, tier, pricing date, context bands, cache evidence, token sums and event counts. Prices remain dynamic. Each request prepares up to 128 pending session-days in the requested range, prioritizing recent days, in four atomic batch statements. Days not yet prepared use the detailed query, so first visits can still be slower. Automatic page polling continues preparation even when an unchanged response returns HTTP 304. Subsequent reads reuse the durable cache. There is no global history lock or partial result.
 
 Database triggers mark only affected days dirty when calls or turns are inserted, corrected, moved or deleted. The next summary request rebuilds those days. Unchanged history stays cached, and session deletion cascades to the cache. A grouping signature protects against changed pricing-context thresholds. Raw fallback remains available during preparation.
 
+Quota forecasting reads exact events directly in batches of at most 8,192 calls, without rebuilding conversation JSON. It preserves event order and invalid-date evidence. A completed page or forecast whose revision changed during reading returns `private, no-store` without an ETag; the quota result also clears its exact revision marker. This prevents frequent Mesh ingestion from continually rejecting completed reads. A later poll validates fresh data. These responses are not transaction-wide snapshots.
+
+See [page loading measurements](page-loading-performance.md) for the measured before/after comparison and `scripts/benchmark-pages.mjs` for reproduction.
+
 Run `node scripts/benchmark-rollups.mjs` for a synthetic 200,000-call comparison with exact result checks. On Node 24, one local measurement was 487 ms without the cache versus 23 ms warm, with 309 ms of initial preparation. This is not a production D1 latency claim.
 
 ## Backups

package.json

@@ -1,11 +1,12 @@
 {
   "name": "codex-usage-dashboard",
   "private": true,
-  "version": "1.6.1",
+  "version": "1.6.2",
   "description": "An independent, privacy-conscious local dashboard for Codex usage, model calls, timing, and theoretical cost.",
   "license": "AGPL-3.0-or-later",
   "type": "module",
   "scripts": {
+    "benchmark:pages": "node scripts/benchmark-pages.mjs",
     "benchmark:storage": "node scripts/benchmark-storage.mjs",
     "test:storage:docker": "node scripts/smoke-relational-docker.mjs",
     "build:ui": "node scripts/build-dashboard-ui.mjs",

public/app.js

@@ -1859,7 +1859,7 @@ async function loadQuotaData(force, source, signal) {
     throw new Error(t("load.errorToast"));
   }
   quotaResponses.delete(key);
-  quotaResponses.set(key, { data, etag: response.headers.get("etag") || cached?.etag });
+  quotaResponses.set(key, { data, etag: response.headers.get("etag") || (response.status === 304 ? cached?.etag : null) });
   while (quotaResponses.size > 3) quotaResponses.delete(quotaResponses.keys().next().value);
   state.data = data;
   render();