Repository navigation
Measure outer test budgets by execution environment #271
Description
Summary
Measure the shared-docs outer check separately on development hosts and hosted runners before deciding whether its 60-second limit should vary by environment. Local validation for the Erlang exporter in #266 reached that limit; the hosted publication audit is a separate workload and remains in progress.
Motivation
The development host has known CPU, memory and disk pressure. The paired local runs failed existing prose-audit or resolver timeouts, so their elapsed times do not isolate the Erlang exporter's cost. A hard cutoff alone cannot show whether the added workload is proportional to the change or whether resource contention dominates.
The Erlang port is preparing its libtmux.org documentation integration. #266 introduces native compiler export; the API-model adapter, guide staging, navigation and rendered-reference checks remain forthcoming. Budget decisions should account for those stages while preserving the existing separation between development checks and the publication audit.
Evidence
Local measurements and separate hosted evidence
The checked-in source loop was invoked with:
$ node scripts/test-loop.mjs outer| Environment | Scope | Whole-command duration | Result |
|---|---|---|---|
| Development host | Checked-in outer loop | 60.954 seconds | FAIL at the enforced cutoff |
| Same development host | Diagnostic completion run excluding only the new Erlang exporter suite | 75.942 seconds | FAIL in existing checks |
| Same development host | Diagnostic completion run including the Erlang exporter suite | 80.988 seconds | FAIL in existing checks; Erlang suite passed |
| Development host | Focused Erlang compiler fixture | 6.226 seconds | PASS |
The diagnostic copies used a 180-second ceiling; the committed 60-second budget was unchanged. The observed difference between the two failed completion runs is 5.046 seconds. This is an inconclusive comparison, not a measured regression: the runs failed, ran sequentially on a shared host and lack resource profiles for causal attribution.
The hosted publication audit runs scripts/test-all.sh. Its Erlang compiler fixture passed in 3.938 seconds. The broader audit was still in progress when this issue was filed; no hosted bottleneck or final outcome has been established. Its duration must not be compared directly with the local outer-loop limit. The contribution guide already states that the publication audit is not limited to 60 seconds.
Proposal
- Record whole-command and phase timings separately for local
outer, hosted checks and the full publication audit, along with CPU, memory, disk pressure and cache state. - Compare equivalent workloads with and without the new compiler fixture, retaining failures and timeouts separately from successful measurements.
- Assess whether a documented environment-specific budget or concurrency adjustment is justified, and bring the measurements back for a maintainer decision.
- Keep the publication audit and development loops distinct when reporting results and when adding the Erlang documentation integration.
References
- feat(api-model[erlang]) Export native compiler documentation #266: draft Erlang compiler exporter and outstanding integration gates.
- Checked-in development-loop budgets.
- Contribution guide: checks and publication audit.
Activity
tony commentedon Nov 8, 2026
The API-model draft in #278 now has a completed hosted audit at
d5c01521448539e2b0b8909171a35e9ccbbfc827. All three checks passed in run 42444598807.test, including setup, publication audit and artifact uploadcheck-publish-previewpublish-preview / publishThe hosted
test-allstep itself took 2h 9m 34s; artifact upload then took 5m 6s. These timings come from GitHub's job and step timestamps. This is the broader publication audit, not the local outer loop, and the draft does not register Erlang routes. These results therefore neither isolate an Erlang regression nor justify changing the 60-second limit. No equivalent-workload baseline or causal resource profile has been established for this run.The committed budgets remain unchanged. Local failures remain open despite the successful hosted audit. The next useful comparison is a phase profile of equivalent local outer-loop workloads at identified revisions and cache states; the hosted publication audit should continue to be reported separately.
tony commentedon Nov 8, 2026
Both official local limits still fail at unchanged draft #278 revision
d5c01521448539e2b0b8909171a35e9ccbbfc827. No source or budget changes were made for this repeat.node scripts/test-loop.mjs innernode scripts/test-loop.mjs outerThe existing resolver corpus failure did not recur in the completed API-model suite. This full outer run still fails, so neither that progress nor the successful hosted publication audit passes the local gate. The site type checker reported zero errors, zero warnings and 30 hints before termination; no final site test count was emitted.
The monitor's before-outer sample showed 8.33% CPU busy and 17.25 GiB available memory. Its final in-run sample showed 59.48% CPU busy and 7.85 GiB available. Samples run every 30 seconds and are host-wide, not a causal profile of this change. The command's child resource counters recorded about 372 CPU-seconds and a largest-child maximum RSS of 4.07 GiB; RSS is not summed across concurrent processes. Existing caches were retained rather than reset.
A source-level lead for the inner loop is its inclusion of the real resolver corpus: the runner selects
resolver.test.ts, whose setup loads ten ports' API catalogs and whose corpus case walks the documentation tree. That test's port list does not include Erlang. This is an existing workload to profile, not proof that it dominates startup or that the Erlang adapter caused the overrun.The next useful change is phase profiling and separating small unit checks from full catalog/corpus work while retaining outer coverage. The 2/10/60-second budgets remain unchanged; these measurements do not establish an environment-specific replacement budget or an isolated Erlang cost.
tony commentedon Nov 8, 2026
Draft #279 adds compiler-derived Erlang type links and native signatures, stacked on #278. The official loops were measured at the same implementation diff without changing budgets:
pnpm test:innercommand.Separately, API-model tests passed in 6.898 seconds, the native-signature renderer tests in 5.392 seconds, API-model type checking in 0.358 seconds, and site type checking in 35.955 seconds. These scoped passes do not waive either whole-loop failure.
The draft now also contains an intentionally failing regression for an independently identified deterministic-header provenance defect. That subsequent test change is marked as a merge blocker, and the earlier API-suite pass is explicitly distinguished from the current failing suite. The source-provenance correction and hosted results remain pending. No Erlang site registration or production deployment is requested.
Resource samples during the whole loops showed CPU activity and memory use rising; they are host-level samples, not attribution of the overrun to Erlang. No timeout was increased, and this is not a matched incremental benchmark against the parent draft.
tony commentedon Nov 8, 2026
Draft #279 now includes the independently reviewed compiler-provenance correction. The native regression file completes in 10.102 seconds; the complete API-model command completes in 13.073 seconds, including those compiler integration cases. These fixtures belong to the outer tier and remain excluded from the official medium loop.
The last whole outer attempt before the final compiler-VM correction still fails at 60.950 seconds. It completes the API-model suite but terminates unfinished site/type-check work at the unchanged deadline. I did not rerun the entire site after the final native-only correction, so current-head whole-site qualification remains open. The earlier medium deadline failure of 10.622 seconds also remains unwaived.
No matched incremental-cost benchmark or replacement budget is established. Passing the native components does not establish a passing whole site. The initial draft's hosted run failed during setup-beam with HTTP 504 before test-all; the new-head run is in progress: https://github.com/libtmux/docs/actions/runs/42469311372.
tony commentedon Nov 8, 2026
Draft #280 adds the reviewed Erlang command-usage reader, stacked on #279. The final source keeps the existing test deadlines.
The complete outer command, including the shared local resource gate, fails at 60.790 seconds. The runner terminates unfinished API/type-check work at the 60-second deadline. This is a failed whole-loop gate, not a passing subset.
Separate final-source checks pass: scanner/derivation regressions in 5.772 seconds; the exact native declaration/call corpus comparison in 12.987 seconds; source-bound attribution audit in 13.081 seconds; script lint in 0.127 seconds; site type checks in 43.535 seconds. The compiler-backed scanner tests belong to the outer tier. These passes do not waive the outer failure.
The first standalone type check under the same 8 GiB, four-core host resource cap fails after 34.748 seconds at Node's automatically selected 2 GiB heap limit. Repeating with a 4 GiB Node heap within that unchanged host cap passes in 43.535 seconds. Both results are retained; no source, test deadline, or host memory cap changed between them.
The earlier candidate's medium-loop deadline failure at 10.624 seconds remains unwaived; it is not a current complete medium pass. Parent #279's hosted publication audit has passed the repaired Rebar3 installation and remains in progress: https://github.com/libtmux/docs/actions/runs/42471187426. Draft #280's hosted checks are also in progress. Neither is hosted timing qualification yet.
No matched parent/candidate incremental benchmark or replacement budget is established. No merge, port registration, production deployment, or deadline increase is requested by this update.
tony commentedon Nov 9, 2026
Draft #292 registers Erlang above the application-manifest reader in #284. The registration uses the qualified core source and retains the existing 2/10/60-second development-loop budgets.
The current local outer gate still fails. Separate completion checks establish useful evidence without replacing that gate:
The static build and rendered-link audit precede the final card-link correction. The correction points to the populated authored Examples overview and passes the existing home-card checks. The remaining multi-port assembly and current-head hosted qualification stay open.
Heavy commands used the shared coordinator's unchanged 8 GiB memory and four-core limits. Queue time is reported separately. Host-level resource samples remain available; these runs do not form a paired parent/candidate benchmark or isolate the incremental Erlang cost. The standalone type-check command already takes longer than the outer deadline under these conditions. Phase profiling and a matched comparison remain the inputs for a budget or concurrency decision.
The first hosted registration run failed during guide staging because the workflow had no Erlang checkout. It did not reach the publication audit. The follow-up checks out the API model's exact source revision, passes it to guide staging, and includes Erlang in the publisher prefix validation. Preview jobs from the first run were skipped; they did not pass.
The follow-up hosted run passed staging and stopped in the destination negative suite. Its committed-data control derived its clock from the bundle's November 3 aggregate date, then rejected the independently refreshed November 8 Erlang records as future data. The fixture now uses today's UTC date for committed records; the synthetic stale/future cases retain fixed clocks. The product verifier and evidence dates are unchanged. The complete local negative suite passes; hosted qualification continues in the current run.
Current source-bound caller checkpoint:
The owning Erlang caller passed exact OTP/Rebar setup, native export and the 376-page version build; publication was SKIPPED. Its build job took 186 seconds including runner setup and artifact upload; the site-build step took 135 seconds. Both downloaded archives match their GitHub digests. Core source is pinned independently from the shared exporter.
The heading probe led to a focused rendered regression: record, function and type headings reproduced the dots, while actual breadcrumb ancestry already passed. The correction passes the relevant naming/heading suites and is finishing type checks. The whole-site candidate audit remains in progress. Draft #292 retains no-preview, so this work publishes no preview.
The local full-source publication audit still fails before assembly because the Python API-model and protocol snapshots require different MCP source commits through one checkout override. No Python evidence or production verifier was rewritten. This is separate from Erlang's explicit MCP exclusion.
The development-loop failures remain unwaived and budgets stay unchanged. These local and hosted observations are different workloads and environments, not a paired measurement of Erlang's incremental cost.