Repository navigation
test(formula): judge each distinct sweep refusal once, and give the validateExpression sweep a measured budget - #30919
Conversation
…alidateExpression sweep a measured budget The validateExpression sweep in expression-refusal-codes.test.ts provokes 3,865 refusals, 199 of them distinct, and re-ran the same assertions on every duplicate. It now judges each distinct refusal once (keyed on the whole refusal), keeps counting every refusal, and still requires every code to be provoked. The breadth of the sweep is its claim and stays. The case is synchronous and took 5034 ms on a contended Test Core (4/8) shard, so it carries an explicit budget of 3x that reading, with the readings in its comment. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8
…mment Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8
📓 Docs Drift CheckNothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs. What this run could not see
Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): |
The two counts in the validateExpression sweep's comments now say where they were read: the 3,865 / 199 refusal count at 98f640d, and the 429-3065 ms band as seven sampled Test Core jobs. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8
|
domain:devx#2 · marchtian · session_017ihL2ebbZkr2QAEG1x7UC8 Contract reviewServed-tier: Inputs read: card #30901 body and both comments (seat claim 6682594819, dev report 6683692239, read as claims); PR #30919 body, labels ( ① Derived judgments(1) No weakened assertion — holds. The key is (2) The timeout — holds. (3) The measurement — holds on spot-check, with one honesty gap. Spot-checked read-only against the Actions API:
(4) Fixes honesty — stands on the card's own ownership clause, with two acts owed by the seat. The card's Wanted says "rebalance the shards, cut the heavy work out of these cases, and/or set explicit per-case timeouts sized from the measurement, each with its reason", then "This card owns the measurement and the formula case." Both owned halves are in the tree and the PR body: the formula case carries a cut and a measured per-case budget with its reason, and the measurement (per-shard walls, the 29-job census, the 31-case default-5 s population) is in the body's acceptance notes with its queries. The shard plan is correctly not edited: (5) The landing — holds. Check-runs on the head at 2026-11-06T11:33:57Z — the gate verdicts, some still running: success — ② Semver levelNone. A test-only diff in ③ Boundary flags
④ Design judgmentThe change holds against the code it cites. Must-fix: none. Should-fix:
Implemented-by: VERDICT: PASS |
Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8
|
domain:devx#2 · marchtian · session_017ihL2ebbZkr2QAEG1x7UC8 Contract reviewServed-tier: Delta review on the first record (comment 6683972205 on this PR, head 4588cae, PASS with three should-fixes). Read: ① Derived judgmentsS1 — the comment readings now name a tree and a sample size; one of the two counts is wrong.
S2 — done. The body's Source line now continues: "The population pools path-filtered S3 — done;
Check-runs on 649a0ab at 11:41:41Z — early: success so far on ② Semver levelNone — unchanged from the first record; comment-only delta. ③ Boundary flagsNone new. One test file; no governed path; ④ Design judgmentThe dedup key and the budget are as judged on 4588cae. The delta is the honesty of the comment that documents the budget: it now carries its tree and its sample size, and the sample size is off by one in the direction that overstates the evidence. Must-fix:
Should-fix: none remaining. S2 and S3 are closed as stated above. Implemented-by: VERDICT: FAIL |
|
domain:devx#2 · marchtian · session_017ihL2ebbZkr2QAEG1x7UC8 Contract reviewServed-tier: Second delta review, on the FAIL record for 649a0ab (comment 6684040945 on this PR) and the PASS record for 4588cae (comment 6683972205). Read: ① Derived judgmentsThe must-fix is closed, and the count is now true. The budget comment reads: "The six other Nothing else changed. The diff is the one comment line and its rewrap;
Check-runs on 3f0ddde at 2026-11-06T11:44:46Z — the run has just started: success — ② Semver levelNone — comment-only delta on a test-only diff; ③ Boundary flagsNone new. One test file; no governed path; ④ Design judgmentAs judged on 4588cae: the dedup key is a complete memo of a deterministic verdict over a four-field value object built in one place, and the budget is per-case, measured, and documented to the repo's own rule. The documentation now carries its tree and a correct sample size. Must-fix: none. Should-fix: none. Implemented-by: VERDICT: PASS |
|
domain:devx#2 · marchtian · session_017ihL2ebbZkr2QAEG1x7UC8 席位核对 — ACCEPT ·
|
Requested by 田騏 · project thread
Fixes #30901
Clause-②: no
Metadata-protocol: none
维护者速读
Test Core (4/8)上跑了 5034 ms,撞上 vitest 默认 5 秒超时变红,下一次推送又绿;本地空闲只要 0.8 秒,是分片负载把它推过了线。packages/formula/src/expression-refusal-codes.test.ts里的refusalKey与VALIDATE_SWEEP_BUDGET_MS注释块,以及扫描用例里judged那几行。pnpm --filter @objectstack/formula exec vitest run src/expression-refusal-codes.test.ts --reporter=verbose,应看到 42 条全过,扫描用例约 0.5–0.7 秒(改前约 0.8 秒)。What changes
packages/formula/src/expression-refusal-codes.test.ts, the case every refusal validateExpression reports on the sweep carries a code from its set, with params the message interpolates (line 870 onorigin/main):validateExpressioncalls (3 roles x 128 inputs x 15 schemas) and gets 3,865 refusals, but only 199 distinct ones. The case ran its five assertions again on every duplicate, about 25,000expectcalls. Every assertion reads only the refusal itself (code,params,message,source, the four fieldsrefusal()invalidate.tsbuilds), so a refusal equal to one already judged would get the same verdict again. It is now keyed withinspect, notJSON.stringify, so anundefinedorNaNparam is still told apart, and judged once.refusalsstill counts every refusal (the> 100control is unchanged), andprovokedstill has to reach every code.VALIDATE_SWEEP_BUDGET_MS = 16_000. It uses the rule frompackages/cli/test/hook-timeout-override-refusal.test.ts: 3x the slowest reading, rounded up to the next second. The slowest reading is the CI failure's 5034 ms, so 3 x 5034 = 15102, rounded up to 16000. The readings and the rule are in the comment above the constant.What was not cut, and why:
buildTypedEnv(cel-engine.ts), which builds a fresh typed CEL environment for each call. That is production code, outside this card's file surface. The inputs are the claim: the docblock says "a new arm reached by an old input is judged", so dropping inputs would weaken the case.beforeAlland no module-scope precompute. The two sweeps share nothing. ThecollectCelRootIdentifierssweep (line 895) walks 40 sources through a different producer and measured 5 to 37 ms everywhere, so it needs neither a cut nor a budget. Moving thevalidateExpressionsweep into a hook would not remove its work. The work would run under the hook's 10 s default, which is a timeout raise by another name.No production file, no shard plan file, no other test file is touched. There is no global
testTimeout, no skip and no retry.Tests
The formula case: before / after, local and CI
origin/main98f640d)Test Corejobs (mainpush / merge group, 2026-11-03..06, a sample of 7 that ran formula)The CI failure was job 127808107135 (
mainpush run 42192379976 ond55acb7828,Test Core (4/8)):Test timed out in 5000ms, case duration 5034 ms. The case is synchronous, so vitest judges it only after the body returns, and 5034 ms is the full run, not a cut-off. In the same run the second sweep took 37 ms.Profile (
--cpu-prof, the sweep x3):validateExpression1189 ms inclusive, of whichbuildTypedEnv830 ms (typeSoundnessIssue,firstTypeMismatch,firstDateTextVerdict). No dynamic import or module load happens inside the clocked window.Mutation: the deduplicated sweep still goes red
Done after the fix was committed (
7a6ff9240d), withnode scripts/ablation-replace.mjs. Inpackages/formula/src/validate.tsthe linereturn { source, message, code, params } as ExprValidationError;was changed to add a paramghost: 'never-in-the-message'tounbound-rootrefusals only.unbound-rootis a code the sweep reaches from only a few of its 5,760 calls.8931039644aa->afa0bba73789(mutation landed)vitest run src/expression-refusal-codes.test.ts -t "validateExpression reports on the sweep": 1 failed,AssertionError: expected 'unbound root \pge` — beyond the recor…' to contain 'never-in-the-message'`8931039644aa),git diff HEADemptyNo build was needed: the test imports
./validatefromsrc.Package suite, typecheck, lint
pnpm --filter @objectstack/formula test: Test Files 65 passed (65), Tests 1703 passed (1703)pnpm --filter @objectstack/formula typecheck: exit 0,check:test-typecheck: OK — 3 file(s) / 7 error(s) / 4 pinned signature(s) held(the ledger is unchanged)pnpm exec eslint packages/formula/src/expression-refusal-codes.test.ts: exit 0filesisdist,README.md,CHANGELOG.md. Afterpnpm --filter @objectstack/formula build,grep -rl 'VALIDATE_SWEEP_BUDGET_MS|refusalKey' distfinds 0 files, and the positive controlvalidateExpressionfinds 4. Soskip-changeset.Acceptance notes
The measurement this card owns. It is a sample, and the queries are named so it can be repeated.
Per-shard wall time, `Test Core (N/8)`, 2026-11-03 10:00Z .. 2026-11-06 10:30Z
Source:
GET /repos/objectstack-ai/objectstack/actions/workflows/224644133/runs?event=push|merge_group&created=>=2026-11-03T10:00:00Z(980 runs: 320mainpush, 660 merge group), thenGET /actions/runs/{id}/jobsfor each completed run. Wall =completed_at - started_atof each successful shard job, setup and build included. The population pools path-filteredmainpush runs with full-plan merge-group runs (sampled push run 42181191624 planned 37 of 84 items, and its shard 1/8 got one item), so a push run's shard reads low. Restrict to full-plan runs before re-deriving the cli slicing from these numbers.Shard 4 runs at 1.53x the mean of the eight medians (776 s). The plan predicts all eight bins equal (
1553/1556/1551/1552/1552/1553/1556/1554sin job 127808107135). The gap comes from the CLI. Shards 1 and 4 each carry one@objectstack/clislice, and the dataset weighs the whole package at 1432.04 s (716 s a slice). In 6 sampled push runs the two slices measured 607-959 s (1/2) and 627-1070 s (2/2), so 1234-2002 s for the package, median 1782 s. Shard 4 also runsplugin-approvals(289 s),verify(340 s) anddriver-memory(183 s) beside the slice. Its formula run is the contended one.The dataset was refreshed 2026-11-02 by
shard-timings-refresh.yml(scheduled, weekly). Its generator needs turbo run summaries that only the scheduled runs keep, so this PR does not touchscripts/test-shard-timings.json, and a hand edit is forbidden. The next scheduled refresh is the place the CLI weight moves.Every 5 s timeout in Test Core over the same window
All 29 failed
Test Core (N/8)jobs were read (GET /actions/jobs/{id}/logs). Ten failed withTest timed out in 5000ms:packages/cli/test/validate-build-gate-parity.test.ts› the command sources use no call shape this scan cannot read, all inTest Core (4/8). This is ci: packages/cli validate-build-gate-parity.test.ts times out at 5000 ms in Test Core (4/8): one main red and one merge-queue ejection in eight hours #30785's case.Test Core (4/8).The other 19 are not timeouts. 18 are
@objectstack/specassertion failures inscripts/liveness/check-liveness.test.tson shards 1 and 2. The 19th is shard-4 job 127654182380, whose log stops in the middle of theverifysuite, so its cause was not measured. #30873's rest case is not among these 29 jobs (its red was on a run outside this window's failed set).The population: cases on the 5 s default that run 2-5 s in CI (sample)
Sample: all 44 successful shard jobs of 6
mainpush runs spread over the window (41615880546, 41887304051, 41996918981, 42086692390, 42136924584, 42181191624), plus the failing job above. vitest prints every case over 300 ms, which gives 3,162 (package, file, case) rows. 402 of them peaked at 2-5 s. Each was classified from its source with the TypeScript AST (theit/testcall's own timeout argument, its enclosingdescribe's, or the package config'stestTimeout): 157 carry a per-case timeout, 16 inherit one from adescribe, 180 sit in a package with a raisedtestTimeout(spec60 s,plugin-auth10 s, ...), 18 areit.eachrows that all carry one. 31 are on the 5 s default. The ones that run closest to the line, beyond #30785's and #30873's:packages/plugins/plugin-hono-server/src/adapter-tls-listener.test.ts› drains on close exactly as the plain listener doespackages/plugins/plugin-hono-server/src/adapter-drain.test.ts› lets an in-flight request complete instead of aborting itpackages/client/src/auth-rotated-session-token.test.ts› login → enable → verifyTotp → disable → deleteUser, no hand-repaired credentialpackages/cli/src/utils/unmanaged-tables.integration.test.ts› the boot this sweep reads from really did compose the deploymentpackages/metadata-protocol/src/reference-sites.derivation.test.ts› two derivations of the same schemas agree exactlypackages/plugins/plugin-security/src/org-admin-assignment-boundary.posture.test.ts› posture 'single' / 'isolated' / 'group': a column the caller's field rules hide is never named in the refusal…packages/cli/test/released-port-production-spawn.pin.test.ts› the census reaches the live tree: every way of spawning the CLI, and the draw, are foundpackages/plugins/plugin-sharing/src/share-link-password.test.ts› a refused upgrade write does not block the read…packages/drivers/driver-sql/src/logger-receiver-detach.test.ts› sql-driver.ts and the modules split out of it contain none of themThe two
plugin-hono-serverdrain cases are not load-driven. They take the same 3.2 s and 4.2 s on an idle box, every run.close()callscloseIdleConnections()while the request is still in flight, so the socket that goes idle after it waits for the client's keep-alive to end. That is a hypothesis from readingadapter.ts, not measured. None is fixed here, because each one sits in another package and outside this claim's file surface. The seat filed them as one class card, #30920, led by the two drain cases. The shard-4/8 reading is on the on-hold shard card #25870 (with the pooling caveat above), for its re-derivation after the weekly refresh.Verification
Gates derived by
node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackfor the final diff, run at ``4588caea1c(the merge oforigin/main`)`:48 commands, each exit code taken before any pipe
--ranreconciliation:✓ dispatch-gates --ran: 48 derived famil(ies) accounted for — 48 run, 0 NOT-MEASURED (a DERIVED zero — all 48 recorded an exit code and none of them is 3).CI: this PR's check-runs are the evidence for every family CI runs.
Test Corereruns the formula suite.🤖 Generated with Claude Code