Skip to content

ci: a weekly merge-queue duration reading against the 22-minute median budget and 32-minute p90 alarm - #35343

Draft
os-warren wants to merge 7 commits into
mainfrom
claude/issue-35322-queue-duration-budget
Draft

os-warren wants to merge 7 commits into
mainfrom
claude/issue-35322-queue-duration-budget

Conversation

@os-warren

Copy link
Copy Markdown
Collaborator

Requested by Warren · project thread

Fixes #35322

Clause-②: no
Metadata-protocol: none
Rule-change: ruled
Ruling-ref: Source: maintainer — Warren, 2026-11-19, review-split discussion thread: 03:16Z 「以上都同意」 (CI speed budget for objectstack; 「变慢两成非常大了」 rejected a 20% allowance); 04:24Z 「1+2+4」, point 2 as amended by the review: budget = median 22 min, alarm = p90 32 min, measured weekly by a script that files a speed-up card when over; a PR's own before/after reading is advisory and never blocks a merge.

维护者速读

  • 改了什么:新增每周一自动量一次合并队列耗时(中位、九成、失败率、最慢的五个作业);中位超过 22 分钟或九成超过 32 分钟就自动开或刷新唯一一张提速卡;AGENTS.md 加两行预算说明。
  • 为什么:落实你 11-19 定的 CI 速度预算。本周实测中位 21.9、九成 31.9 分钟,已贴着上限,没人盯就会悄悄变慢。
  • 最坏情况与能否撤回:量错会多开或漏开一张提速卡,不挡任何合并、不碰任何必需检查;删掉这个 workflow 即可撤回。
  • 需要你做的:AGENTS.md 的行数上限是 1050,这两行让它到 1052(该门禁现在是红的)——请一句话批准上调 2 行并给出可引用的评论,或指定删哪两行;之后审批合并(改了 workflow 和 AGENTS.md,属人合面)。

What changes

  • scripts/ci/queue-duration.mjs (new, read-only against GitHub). Lists every merge_group run created in the window (--since 7d|24h|ISO, --until), groups by head SHA, and for each SHA whose every queue-workflow run concluded green measures the first run's created_at to the last run's updated_at (waiting for runners and re-runs included). Prints p50 / p90 / max (nearest rank), the SHA counts (green, red, cancelled, still running), the queue failure rate (SHAs with a red run over completed SHAs), which workflow finished last, and the five slowest jobs of that critical run over an evenly spaced sample of 40 green SHAs. --repo, --json, --render FILE, --via gh (a seat whose egress only gh reaches). Exit 2 = NOT MEASURED: a window that cannot be listed whole is split in halves down to one hour and then refused, never truncated; a sample under the budget's floor is not judged.
  • scripts/ci/queue-budget.json (new): budgetMedianMinutes: 22, alarmP90Minutes: 32, minSample: 20, the baseline with its window, and the five queue workflows (CI, Governed Surface Guard, Lint & Type Check, PR-Body Guards, Spec Liveness Check, the five files with a merge_group: trigger). Its note and the script header say changing a number is the maintainer's call. A merge_group workflow not on the list is warned about in the reading, never silently folded in.
  • .github/workflows/queue-duration-budget.yml (new). Weekly, Monday 07:09 UTC, plus workflow_dispatch, plus a pull_request trigger paths-filtered to its own three files. Job measure (actions: read, contents: read, 20 min): self-tests the module, reads objectstack, reads hotcrm report-only (continue-on-error, no card), writes both readings to the run summary, keeps them as an artifact, and reds only on NOT MEASURED (the patrol reporting its own death). Job file (contents: read, issues: write, 10 min) runs only when the verdict is over, the event is not pull_request and the ref is main; it calls fileQueueCard: one card, fixed title merge queue over its speed budget plus the plain-text body marker os-queue-duration-budget, a bounded open-issue scan that refuses rather than files when it cannot establish absence, the oldest match refreshed in place, labels tooling, area:fleet, domain:devx, priority:p2, pm:queue and issue type Task on create only (the type is its own write, so a refused type costs the type, never the card).
  • Why its own workflow, not a job in fleet-patrols.yml: that file's pull_request: paths list is frozen as a set by scripts/check-platform-checklist-watchdog.mjs, and a pull_request run there exercises every job. Exercising this reading on its own PRs would mean widening the frozen list and editing the pin and its self-test, and every PR touching a sibling patrol's input would then also spend a live queue reading. A separate file leaves the pin untouched and scopes the PR run to this reading. No merge_group: and no pull_request_target: trigger; not a required context.
  • .github/workflows/lint.yml: one run_self_test line in the existing "Moved workflow step bodies self-test" collector, so Lint & Repo Gates runs the new self-test on every PR.
  • AGENTS.md Build & Test: two lines stating the budget file, the weekly read, the one speed-up card, and that a CI-changing PR reports its job's before/after median as advisory.

Why

The maintainer set a CI speed budget for objectstack (Ruling-ref above). Nothing watched the queue's duration trend, so a slower queue was noticed by nobody. A PR's own before/after reading stays advisory: a gate grading a duration reports and never reds a required job (AGENTS.md Build & Test).

Verification

All readings below are against objectstack-ai/objectstack; script at branch commits fe60502e..1397a8ab (the measuring code unchanged across them), origin/main at a998964d when the readings were taken.

Self-test: 53 assertions, 8 floored batteries

node scripts/ci/queue-duration.mjs --self-test on head 1397a8ab:

✓ queue-duration --self-test: 53 assertions across 8 floored batteries.

Batteries: percentile and window parsing (6) · grouping by head SHA (9) · verdict, each threshold alone, sample floor, report-only (6) · listing: the adaptive split, cross-slice de-dup, a failed page fails the listing (5) · critical-run jobs (4) · budget file, the tracked file plus six malformed shapes (7) · the card (10, including the de-dup proof: a second run sees the first run's card, refreshes it and never files a second) · rendering (6). Roster, battery floor and verdict handshake per AGENTS.md "Writing a --self-test".

Live dry run, last 7 days: p50 21.9 · p90 31.9 · max 55.6 min, WITHIN

node scripts/ci/queue-duration.mjs --via gh --since 7d, window 2026-11-12T04:42:36Z .. 2026-11-19T04:42:36Z, 165 API reads:

  • 10195 runs · 2039 head SHAs · 1802 all-green · 224 red · 12 cancelled · 1 still running
  • all-green SHAs: p50 21.9 min · p90 31.9 min · max 55.6 min, verdict WITHIN (median 21.9 ≤ 22, p90 31.9 ≤ 32)
  • queue failure rate: 224 of 2038 completed SHAs had a red run (11.0%)
  • the run that finished last: CI on 1801 of 1802 green SHAs
  • slowest jobs of that run (median over 40 sampled SHAs): Test Core (9/10) 24 min (3 sampled), Test Core (7/8) 19 min, Test Core (8/8) 18.5 min, Test Core (3/10) 17.3 min, Test Core (3/8) 17.2 min. Both shard spellings appear because the Test Core shard count changed inside the window.

Against the card's baseline window (--since 2026-11-17T00:00:00Z, to 2026-11-19T04:39Z): p50 21.6 · p90 31.5 · max 51.3 min over 456 all-green SHAs, against the baseline's 21.7 / 31.9 / 34.5 over 176. The percentiles land within half a minute. The sample size and the max differ, and the baseline's record does not say why its count is 176; this reading counts every all-green SHA in the window, re-runs included by design, and its max is a single SHA.

hotcrm, report-only

--repo objectstack-ai/hotcrm --since 7d: 46 runs (CI only) · 45 all-green · 1 red · p50 24.1 · p90 40 · max 41.8 min; slowest job CI / Build and Test (22.x) 24.1 min. Printed to the run summary only; no card, never judged against the budget.

Local gates

Derived with node scripts/pm/dispatch-gates.mjs --commands (59 commands; tree objectstack-ai/objectstack). On the final head 1397a8ab: 57 of the 59 run directly, 56 exit 0, 1 exit 1 (below); pnpm check:pm-dispatch-gates is the 58th; pnpm check:type-check-debt is NOT MEASURED. dispatch-gates --ran with every exit code recorded: 59 derived famil(ies) accounted for — 58 run, 1 NOT-MEASURED.

  • pnpm check:pm-dispatch-gates (fast tier, 2249 cases) green on 5b3f27f4. Its first run, on an earlier head, caught two things this PR then fixed: the new paths-filtered workflow discovered no check family (now the measure job's self-test step is its family), and a no-check-families marker would have broken the marker census.
  • pnpm check:workflow-status-functions caught the file job's if: reading needs.*.outputs.* with no status function; it is now !cancelled() && ....
  • ⛔ Red, on purpose, and the one open question: pnpm check:pm-skill-ratchet — AGENTS.md is 1052 lines; the ratchet ceiling is 1050. The two lines are the card's item 4. Raising the CEILINGS row needs the ruling comment id that orders the text, and this card's ruling lives in a discussion thread with no GitHub comment id, so the row is left at 1050. Both lines fit the 120-byte line budget.
  • pnpm check:type-check-debt: NOT MEASURED locally (timed out at 300 s on the shared box); the diff touches no package. CI runs it.
  • Not run locally, CI-owned: pnpm lint, test:repo, the type-check lanes.
  • workflow_dispatch: NOT MEASURED — a dispatch only runs a workflow that is on the default branch. This PR's own pull_request run exercises the measure job live and writes nothing.

Acceptance notes

  • Changeset: none. Nothing here ships in any package's files[] (scripts/ci/**, .github/workflows/**, AGENTS.md), so the PR wants the skip-changeset label, which this session could not write.
  • The card is filed only on an over week. A week back within budget writes nothing, so an open speed-up card is closed by the seat that lands the speed-up, not by this workflow.
  • The job wait column (job created_at to started_at) reads 0 minutes across the sample, so that column does not capture runner waiting on this repository; the waiting is inside each SHA's first-start-to-last-end span, which is what the budget grades.

Human Minimum

  • 1. Schemas start as Zod; packages/spec holds no runtime logic. (not touched)
  • 2. Every name I added uses the casing for its kind.
  • 3. No formatter ran; pnpm lint passes. (no formatter ran; pnpm lint left to CI)
  • 4. One issue, one branch off an up-to-date main, one pull request.
  • 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. (publishes nothing; skip-changeset)
  • 6. packages/spec touched: its generated artifacts are regenerated and committed. (not touched)
  • 7. No hand edits to content/docs/references/, packages/*/CHANGELOG.md or content/docs/releases/.
  • 8. Title, body and comments are in English.
  • The Before Submitting list in CONTRIBUTING.md is done. (not read by this session)

Generated by Claude Code

… AGENTS.md budget sentence

Claude-Session: https://claude.ai/code/session_01JQwxbYzM6act7DgRZuPrxu
Co-authored-by: Claude <noreply@anthropic.com>
…name the wait column precisely

Claude-Session: https://claude.ai/code/session_01JQwxbYzM6act7DgRZuPrxu
Co-authored-by: Claude <noreply@anthropic.com>
… AGENTS.md sentence to two lines

Claude-Session: https://claude.ai/code/session_01JQwxbYzM6act7DgRZuPrxu
Co-authored-by: Claude <noreply@anthropic.com>
…h is also its discovered check family

Claude-Session: https://claude.ai/code/session_01JQwxbYzM6act7DgRZuPrxu
Co-authored-by: Claude <noreply@anthropic.com>
@os-warren os-warren self-assigned this Nov 19, 2026
@github-actions github-actions Bot added size/xl ci/cd documentation Improvements or additions to documentation labels Nov 19, 2026
@os-warren os-warren added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Nov 19, 2026 — with Claude
…docs/agents/build-and-test.md

AGENTS.md returns byte-identical to the merge base; the explanation lives
on the Build & Test companion page.

Claude-Session: https://claude.ai/code/session_01JQwxbYzM6act7DgRZuPrxu
Co-authored-by: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd documentation Improvements or additions to documentation size/xl skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: a weekly merge-queue duration reading against a 22-minute median budget and a 32-minute p90 alarm, filing a speed-up card when over

2 participants