Skip to content

fix(scenarios): the run grader reads a hook's declared refusal off the door's declaredCode - #33556

Queued
objectstack-fleet[bot] wants to merge 1 commit into
mainfrom
claude/project-thread-rl158j
Queued

objectstack-fleet[bot] wants to merge 1 commit into
mainfrom
claude/project-thread-rl158j

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Requested by Leehom · project thread

domain:devx#4 · os-project-manager · session_01FEkQ5nw3MDkRymEzv6FDJs

Fixes #33497

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:场景 runner 把 MCP 门返回的 declaredCode(hook 自己声明的拒绝码,被门降级到 code 旁边)保留到 ObservedAnswer 上;评分器 refusalMeets 读它:声明码等于 declaredCode 时精确命中,声明 VALIDATION_FAILED 时 4xx 上带 declaredCode 即算 app 自己的拒绝。
  • 为什么改:门实际返回的是 RESOURCE_CONFLICT 409 + declaredCode: ASSET_NOT_IN_STOCK,runner 只留了 code,评分器看到的是平台冲突码,于是 app 正确拒绝的 check_out_asset_in_repair 被判失败;原测试把 hook 码直接塞进 code,门从来不这么返回。
  • 风险与回滚:只改评分器和 runner,门、线上形状、卡片期望都没动;平台码仍然严格:没有 declaredCode 的目录码不命中别的声明码,5xx 或无 status 的 declaredCode 不命中。回滚即还原本 PR。
  • 先看:packages/qa/scenarios/src/score/scenario-operation-grade.ts 的 refusalMeets(L225)和 refusalText(L237);packages/qa/scenarios/src/runner/observe.ts 的 declaredCodeOf(L218)、refusedBy(L242)。
  • 两分钟验证:pnpm --filter @objectstack/scenarios exec vitest run test/runner/mid-tier-run-readings.test.ts,预期 49 条全绿。
  • 需要你做的:无,席位落地。

What changes

  • Runner, packages/qa/scenarios/src/runner/observe.ts. refusedBy (L242) now takes the tool output and returns the refused answer with code, status and declaredCode, each only when served. declaredCodeOf (L218) reads declaredCode off the same envelope refusalOf reads (the same two JSON readings; the nested error object when it carries a string code, else the flat body), and answers undefined for an absent or empty one: no default is invented. The three refused-answer paths go through it: a write call's refusal (refusalAnswerOf), the last read on the object when no write was made (lastReadRefusalOutput, L147, read at L270), and the runner's probe (observeProbe builds a one-call turn and reads it through observedAnswer).
  • Type, scenario-operation-grade.ts L94. ObservedAnswer's refused member gains the optional declaredCode?: string.
  • Grader, refusalMeets (L225). (a) A declared code equal to the answer's declaredCode meets exactly, like code === declared. (b) For a declared VALIDATION_FAILED, a present declaredCode at a 4xx status meets, whatever the catalogue code beside it. A catalogue code with no declaredCode still never meets a different declared code, and a declaredCode at 5xx or with no status does not meet.
  • refusalText (L237) names what met: refused RESOURCE_CONFLICT (409), the hook's declared refusal ASSET_NOT_IN_STOCK.
  • Docs. The refusalMeets docblock and packages/qa/scenarios/README.md L70–L75 now describe the wire shape: code is the status's catalogue code, declaredCode is the hook's own.

The wire codes the tests pin are the door's (HttpStatusErrorCodeMap, packages/spec/src/api/errors.zod.ts L183, with 4xx otherwise VALIDATION_ERROR through standardErrorCodeForHttpStatus): 400 and 422 → VALIDATION_ERROR, 403 → PERMISSION_DENIED, 404 → RESOURCE_NOT_FOUND, 409 → RESOURCE_CONFLICT, 500 → INTERNAL_ERROR.

Tests

All in packages/qa/scenarios/test/runner/mid-tier-run-readings.test.ts.

  • Item 2, re-pinned on the wire shape (L162). The hook's OVER_RECEIPT at 400, 403, 404, 409 and 422 is served as the status's catalogue code with declaredCode: 'OVER_RECEIPT', and meets VALIDATION_FAILED. Negatives: each catalogue code with no declaredCode (VALIDATION_ERROR 400, PERMISSION_DENIED 403, RESOURCE_NOT_FOUND 404, RESOURCE_CONFLICT 409, INTERNAL_ERROR 500) does not meet; declaredCode at 500 does not meet; declaredCode with no status does not meet; a hook's NOT_REQUESTER served at 400 does not meet PERMISSION_DENIED.
  • Rule (a), new (L184). RESOURCE_CONFLICT 409 + declaredCode: 'ASSET_NOT_IN_STOCK' meets a declared ASSET_NOT_IN_STOCK (and RESOURCE_CONFLICT); the same conflict with no declaredCode does not meet ASSET_NOT_IN_STOCK; a different declared code (OVER_RECEIPT) does not meet.
  • "Applies to every card" now grades each card's validation operation on VALIDATION_ERROR 400 + declaredCode, with the detail refused VALIDATION_ERROR (400), the hook's declared refusal ….
  • R1.8 (L647) grades check_out_asset_in_repair on RESOURCE_CONFLICT 409 + declaredCode: 'ASSET_NOT_IN_STOCK', both halves passing with the new detail; RESOURCE_CONFLICT 409 with no declaredCode fails.
  • Item 4's "an app's own code is no permission refusal" probe is served as VALIDATION_ERROR 400 + declaredCode: 'NOT_APPROVER'.
  • 2b, new runner tests (L214), on asset_lifecycle check_out_asset_in_repair:
    • declaredCode reaches the ObservedAnswer through a create_record call and through the probe, in the nested ({ error: { code, message, status, declaredCode } }) and the flat ({ error, code, status, declaredCode }) shape;
    • through the last read on the object when no write was made;
    • absent stays absent: a door that served none, or an empty one, leaves no key (toStrictEqual);
    • end to end through gradeCardOperation: the served shape passes both halves; the same conflict with no declaredCode fails agent:refused:VALIDATION_FAILED-or-withheld and platform:refused:VALIDATION_FAILED.

Ablation (head 06330bd9, scripts/ablation-replace.mjs, anchor ...(declaredCode !== undefined ? { declaredCode } : {}), in observe.ts deleted, anchor 1 → 0, blob d6a5337a5748 → d9534323aac4; the tests import src/, so no dist is in the path):

Ablated: 3 failed | 46 passed (49); restored: 49 passed
× reaches the ObservedAnswer through a write call and through the probe, nested and flat
× reaches it through the last read on the object when no write was made
× grades end to end: the served wire shape passes both halves; ⛔ the same conflict with no declaredCode fails both
    + "agent:refused:VALIDATION_FAILED-or-withheld",
    + "platform:refused:VALIDATION_FAILED",
Test Files  1 failed (1) · Tests  3 failed | 46 passed (49)
ablation-replace: ok restored: blob == HEAD (d6a5337a5748) and `git diff HEAD` is empty

The "absent stays absent" case stays green under the ablation, as it should.

Acceptance notes

  • Legacy shape: no test kept. No door the runner calls serves a hook's unregistered code as code. The /data faces of the MCP record tools answer through refusalAsTheDoorServesIt (packages/mcp/src/mcp-http-tools.ts L990), which serves the door's catalogue code with declaredCode whenever the door demoted (dataDoorRefusalEnvelope, L954); the resource body does the same (packages/mcp/src/mcp-server-runtime.ts L2179–L2187). The door's demotion is resolveThrownHttpError's code row (packages/types/src/thrown-http-error.ts L253: a registered code stays, anything else is derived from the status) with demotedDeclaredCode (L526). run_action demotes the same way (packages/types/src/actions-door-answer.ts L58, L69–L70). So the legacy code: 'OVER_RECEIPT' and code: 'ASSET_NOT_IN_STOCK' cases are dropped.
  • Observed, not changed: the legacy branch. refusalMeets still lets a code outside the ErrorCode catalogue at 4xx meet a declared VALIDATION_FAILED (the branch before this change). It is kept so that no reading narrows beyond the order's rulings, and it is now unpinned because no door was found serving that shape. class: observation · reach: none · carrier: none. The seat may retire it on a later card.
  • Observed, not changed: a hook refusal at 403. A hook that refuses at 403 reaches the grader as code: 'PERMISSION_DENIED' with its declaredCode. It meets a declared PERMISSION_DENIED through code === declared, as before this change, and now also a declared VALIDATION_FAILED through rule (b) (ruling A: any 4xx). class: observation · reach: none (no card declares PERMISSION_DENIED for a hook's own refusal) · carrier: none.
  • refusalOf left as it is. It lives in packages/qa/scenarios/src/score/scenario-run-grade.ts, outside the ordered surface, so declaredCode is read in observe.ts by a private reader that mirrors its envelope selection, rather than by widening refusalOf's Refusal.
  • The new optional field ObservedAnswer.declaredCode is an additive optional member of a published package type, ordered under Clause-②: no.
  • No live door probe. The card quotes PR docs(skills): a hook that enforces a business rule declares refusals and refuses by code #33488's dev's probe of the served shape; the tests pin that shape.

Verification

On head 06330bd9 (branch claude/project-thread-rl158j, merge base origin/main e3f0cfd4), each exit code captured before any pipe:

  • pnpm --filter @objectstack/scenarios test: exit 0, Test Files 32 passed (32), Tests 712 passed (712).
  • pnpm --filter @objectstack/scenarios typecheck (tsc --noEmit, which compiles the tests): exit 0. pnpm --filter @objectstack/scenarios build: exit 0.
  • node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios --no-inline-config: exit 0 (the package has no lint script).
  • Derived gates (node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack, 59 families): 58 exit 0; pnpm check:dual-build-cjs-loads exit 3 PREREQUISITE NOT MET (needs every package's dist/), NOT MEASURED, CI runs it. --ran: "59 derived famil(ies) accounted for — 58 run, 1 NOT-MEASURED".
  • dispatch-gates --tier: no path-derived mandate. 239 changed lines, under the 3000 threshold. No governed surface.
  • Changeset .changeset/33497-scenario-grader-declared-code.md (file name unique to this card).
  • CI families are read from this PR's check-runs, not re-run here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FEkQ5nw3MDkRymEzv6FDJs

…e door's declaredCode

The MCP door demotes a hook's own refusal code to `declaredCode` and serves
the status's catalogue code on `code`. The runner kept only `code` and
`status`, so the grader read the hook's ASSET_NOT_IN_STOCK at 409 as a
platform RESOURCE_CONFLICT and failed a declared VALIDATION_FAILED.

A refused ObservedAnswer now carries `declaredCode` when served, and
refusalMeets reads it: a declared code equal to it meets exactly, and a
declared VALIDATION_FAILED is met by one at any 4xx.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FEkQ5nw3MDkRymEzv6FDJs
@github-actions github-actions Bot added size/m documentation Improvements or additions to documentation tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface labels Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/scenarios, touching 15 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/qa/scenarios/README.md), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

3 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/automation/hooks.mdx (via declaredCode (symbol, a field of type ObservedAnswer))
  • content/docs/permissions/system-context.mdx (via describeEffectiveAccess (sdk, the bare tail of client method security.describeEffectiveAccess, bound to GET /api/v1/security/users/:id/effective-access))
  • content/docs/protocol/kernel/error-handling.mdx (via declaredCode (symbol, a field of type ObservedAnswer))

⛔ 2 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17/17-1.mdx (via declaredCode (symbol, a field of type ObservedAnswer))
  • content/docs/releases/v18/18-4.mdx (via describeEffectiveAccess (sdk, the bare tail of client method security.describeEffectiveAccess, bound to GET /api/v1/security/users/:id/effective-access))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/qa/scenarios/README.md) — pages documenting those are invisible to this run
  • the SDK route bridge reached 65 of 247 client-bound route-ledger rows — the other 182 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 182: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 59 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 123 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 1 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e3f0cfd422d69100448ccd4f6a329c90a25715f7 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 9b77b1aa9caeffc7922f9dd71182dea61bea613a — the merge of head 06330bd9bf55a195473553a36c959f3b21a58265 into base e3f0cfd422d69100448ccd4f6a329c90a25715f7, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 9b77b1aa9caeffc7922f9dd71182dea61bea613a && git checkout 9b77b1aa9caeffc7922f9dd71182dea61bea613a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e3f0cfd422d69100448ccd4f6a329c90a25715f7 06330bd9bf55a195473553a36c959f3b21a58265 && git checkout -B drift-repro e3f0cfd422d69100448ccd4f6a329c90a25715f7 && git merge --no-ff 06330bd9bf55a195473553a36c959f3b21a58265

node scripts/docs-audit/affected-docs.mjs --json e3f0cfd422d69100448ccd4f6a329c90a25715f7

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e3f0cfd422d69100448ccd4f6a329c90a25715f7 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:devx#4 · os-project-manager · session_01FEkQ5ooRyyS95kdyC2dPEX

席位核对 · ACCEPT · head 06330bd9 · 2026-11-15T00:41Z

  • 形态: draft → main, attribution first, Fixes #33497. Paths from the PR's file list: .changeset/33497-scenario-grader-declared-code.md (+12/-0), packages/qa/scenarios/README.md (+6/-3), packages/qa/scenarios/src/runner/observe.ts (+52/-9), packages/qa/scenarios/src/score/scenario-operation-grade.ts (+31/-15), packages/qa/scenarios/test/runner/mid-tier-run-readings.test.ts (+86/-25). No governed path, no packages/spec/src/**, Clause-②: no ⇒ no contract-review face; lane seat lands.
  • Diff read: The runner keeps the door's declaredCode on a refused ObservedAnswer (write calls, the last read and the probe; nested and flat envelopes, the same two readings as refusalOf), absent when unserved. refusalMeets: (a) declaredCode === declared meets exactly; (b) for a declared VALIDATION_FAILED, a present declaredCode at a 4xx status meets whatever the catalogue code; a catalogue code with no declaredCode stays strict, and a declaredCode at 5xx or with no status does not meet under (b). refusalText names the hook's declared refusal. Tests re-pinned on the wire shape (RESOURCE_CONFLICT 409 + declaredCode); the runner pass-through ablated 3 of 49 red, restored. Docblock and README corrected.
  • 门禁 check-runs (head 06330bd9): 49 runs · 33 success · 16 skipped · 0 not green; Lint & Repo Gates success, TypeScript Type Check success. K = 6 merges behind main (< 15).
  • Changeset: one patch changeset for @objectstack/scenarios (.changeset/33497-scenario-grader-declared-code.md, file name unique to this card), with the Clause-②: no / Metadata-protocol: none lines.
  • 接受的偏差: Two noted in the PR's Acceptance notes, both accepted as is: the old branch (an app code outside the catalogue at 4xx) stays with no door serving it and no pin, which is harmless and keeps any non-MCP door's own code meeting; a hook refusal at 403 also meets VALIDATION_FAILED under (b), which is the ordered 4xx reading. check:dual-build-cjs-loads was NOT MEASURED locally (exit 3); CI covers it.
  • 范围外去向: The docs-drift bot lists three hand-written pages naming declaredCode; they describe the door's wire shape, which this PR does not change, so none is owed. Skills-docs: not owed (Metadata-protocol: none).
  • 落地条款: non-governed, all checks green ⇒ the seat readies and arms auto-merge. Ruling-ref: the maintainer's standing authorisation 「按照 pm-dispatch 规则复核过的都授权」 (2026-10-25T11:35Z) and 「你应该使用 pm-dispatch 技能审核和落地pr」 (this account's project).

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review November 15, 2026 00:41
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Nov 15, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qa(scenarios): refusalMeets never sees a hook's declared refusal; the MCP door serves it as declaredCode beside the status's catalogue code

1 participant