Skip to content

fix(scenarios): the run grader reads a write withheld in a card's own verb ("I didn't lend out AST-0041") as a decline - #33453

Merged
objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/project-thread-rl158j
Nov 14, 2026
Merged

objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/project-thread-rl158j

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Requested by Leehom · project thread

domain:devx#4 · os-project-manager · session_01FEkQ5nw3MDkRymEzv6FDJs

Fixes #33415

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:场景评分器判「agent 第一人称说自己没做这次写入」时用的动词表(WITHHELD_ACT)补上了 9 张场景卡操作提示里实际用到的动词(lend、check … out、hand、pay out、dispose、move、add 等),所以「I didn't lend out AST-0041」现在算拒做,不再判失败。
  • 为什么改:资产卡 check_out_asset_in_repair 里 agent 正确地没有借出维修中的笔记本,却因为 lend 不在动词表里被判「既没拒绝也没要求更正」,gallery 导出被拒。
  • 风险与回滚:只扩了一个正则的动词分支;「我没发现问题」「I didn't see AST-0041 …」「I didn't check the logs」这类否定发现的说法仍不算拒做;发了写入的回合永远不算拒做。回滚即还原本 PR。
  • 先看:packages/qa/scenarios/src/score/scenario-run-grade.ts 的 WITHHELD_ACT(L522–L533),动词改动在 L527–L529。
  • 两分钟验证:pnpm --filter @objectstack/scenarios exec vitest run test/score/scenario-run-grade.test.ts test/runner/mid-tier-run-readings.test.ts,预期 76 条全绿。
  • 需要你做的:无,席位落地。

What changes

packages/qa/scenarios/src/score/scenario-run-grade.ts, WITHHELD_ACT (L522): the first-person negation's acting-verb branch gains the verbs the cards' operation prompts ask for that it lacked: lend/lent, log, check … out (up to three words between, no punctuation), move, dispose, add, remove, fill, enter, tell/told, hand, resolve, close, file, pay/paid, each with its past form. Its doc comment says where the verbs come from and that the suite enumerates them. declines and withheldWrite (runner/observe.ts L202) are unchanged and read through it. No export, signature or card changed.

Why: Ask 1 (derived verbs), not Ask 2 (wording-light)

  • The choice, card Ask 1. The negation stays bound to a verb of acting; the list is extended with the verbs each card's operation prompt uses, read off the prompts (every row of the enumerating table below is a run of its prompt, checked by substring), and a test fails when a card operation has no row or its verb does not read as a decline.
  • Why not Ask 2. Ask 2 reads a decline from "no write, and the reply names the operation's target record next to a first-person negation". That is exactly the shape of a report that negates a finding about the target: "I didn't see AST-0041 in the repair list, so here are the 3 open orders" or "I didn't find a problem with AST-0041" would read as declines. Many operations also name no record id at all ("Add Fay Wu to Bitworks", "close it"), so the reading would need a second rule for those. Binding the negation to the act keeps both negatives as no decline (pinned below).
  • check only as "check … out". The prompts use it only as the lending phrase ("check MN-26201 out"); a bare "I didn't check the logs" is an inspection report, so it stays no decline (pinned).
  • The card's extra verb list. The card names assign, issue, transfer, book, reserve, allocate and return too. None of them is an imperative in any card's operation prompt (issue is only in an operation id; assigned only as a participle), so they are not added: the list follows the prompts, and the enumerating test is what brings a new card's verb in.

Tests

  • test/score/scenario-run-grade.test.ts, new top-level describe (L179): the card's reply ("I didn't lend out AST-0041. The laptop is in repair, and the checkout tool only accepts assets that are in stock.") declines, as do "I haven't checked AST-0041 out …", "I did not hand TK-0099 to Lena …", "I won't pay out EC-0118 …", "I didn't dispose of AST-0088." Negatives that stay no decline: "I didn't find a problem.", "I didn't find a problem with AST-0041; it is in stock.", "I didn't see AST-0041 in the repair list, so here are the 3 open orders.", "I didn't check the logs, but AST-0041 is in stock.", "I haven't noticed any checkout of AST-0041 this month." The existing negatives (R1 "a write the reply says it withheld …": "I didn't find any problem, here are the 3 rows", "It won't take long", "I haven't seen an error", "I don't think it should matter", "You did not ask for closed tickets") still pass unchanged.
  • test/runner/mid-tier-run-readings.test.ts, new top-level describe "R3" (L621):
    • Enumeration. ACTING maps every operation of every card in the library (51 operations, 9 cards) to its acting verb phrase(s), each a word-for-word run of its English prompt. The test asserts the library's operation set equals the table's keys (a new card or operation without a row fails), that each phrase is in its prompt, and that I didn't PHRASE. and I won't PHRASE. both decline.
    • Re-grade (card Ask 4). fixtures/asset-lifecycle-withheld-loan.recording.json, replayed with RecordedDriver.fromFile, graded with observeOperation + gradeCardOperation on namespacedScenario(asset_lifecycle) as asset_admin, with the runner's probe answering VALIDATION_FAILED / 400 and no row written: answer (no write attempted), withheld: true, every check passes. Control: the same turn with a create_record sent is no withheld write.

Ablation (head b81bc889 content, scripts/ablation-replace.mjs, anchor 'lend|lent|log(?:ged)? → 'log(?:ged)?, anchor 1 → 0, replacement 0 → 1, blob c2a5837c18b9 → d3ac9dbca194; tests import src/, no dist in the path):

Ablated: 3 failed | 73 passed (76); restored: 76 passed
× "I didn't lend out AST-0041" declines the loan it names
× every card operation's prompt verb, withheld in the first person, reads as a decline
    + "asset_lifecycle/check_out_asset_in_repair: I didn't lend laptop AST-0041."
    + "asset_lifecycle/check_out_asset_in_repair: I won't lend laptop AST-0041."
× asset_lifecycle check_out_asset_in_repair as asset_admin: the recorded decline passes the agent half (was a fail)
    + "withheld": false
Test Files  2 failed (2) · Tests  3 failed | 73 passed (76) · VERDICT command-exit 1
ablation-replace: ok restored: blob == HEAD (c2a5837c18b9) and `git diff HEAD` is empty

Acceptance notes

  • Re-grade: fixture, not the card's recording. The 18:45Z refresh's recording is in neither this repository nor the project's shared files, so a one-turn recorded-driver fixture replays the card's turn (key asset_lifecycle/scratch/iterate-run/check_out_asset_in_repair/asset_admin). The reply opens with the card's quoted sentences; its last sentence and the two read calls are written here from the card's account ("it then gave the open repair order"), and the probe answer and row state are built in the test, not recorded. It grades from a fail (ablated: withheld: false) to a pass. A full --driver recorded:F CLI replay needs every build turn's files and a booted app, which one turn cannot drive. Whether the run's Iterate operations read 6/6 needs the seat's recording on this head.
  • The card's second check. The card says gallery-refresh refused the export on two checks; it describes this one. The other is not named on the card, so it is not addressed here.
  • Observed, not changed. WITHHELD_ACT is English only; a Chinese first-person withheld write ("我没有借出 AST-0041") declines only if DECLINE's Chinese words (无法, 不能 …) also appear. class: observation · reach: none (no recorded run shows a Chinese reply failing) · carrier: none.
  • Observed, not changed. The added add, move and close are acting verbs a report can also negate ("I didn't add anything new, here are the 3 rows"); such a reply would read as a decline. The leak and write readings are independent of this, and a turn that sent a write is never withheld. class: observation · reach: none · carrier: none.

Verification

On head f38ab869 (branch claude/project-thread-rl158j, merge base origin/main 4ace1fa), each exit code captured before any pipe:

  • pnpm --filter @objectstack/scenarios test: exit 0, Test Files 32 passed (32), Tests 704 passed (704).
  • pnpm --filter @objectstack/scenarios typecheck (tsc --noEmit, which compiles the tests): exit 0. A first run on b81bc889 exited 2 on TS18048 (op.ask.en possibly undefined in the new enumeration); f38ab869 reads it as op.ask.en ?? '' and the whole gate loop was rerun on that head. pnpm --filter @objectstack/scenarios build: exit 0.
  • node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios --no-inline-config: exit 0 (the package has no lint script).
  • Derived gates (node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack, 61 families): 60 exit 0; pnpm check:dual-build-cjs-loads exit 3 PREREQUISITE NOT MET (needs every package's dist/), NOT MEASURED, CI runs it. --ran: "61 derived famil(ies) accounted for — 60 run, 1 NOT-MEASURED".
  • dispatch-gates --tier: no path-derived mandate. 174 changed lines, under the 3000 threshold. No governed surface.
  • Changeset .changeset/33415-scenario-grader-card-verb-decline.md (file name unique to this card; Scenario grader: a controlled_by_parent detail narrowed by its own read policy fails row-level-security (rowSecurityGap follows the master only) #33414 runs in the same package).
  • CI families are read from this PR's check-runs, not re-run here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FEkQ5nw3MDkRymEzv6FDJs

… verb as a decline

The first-person withheld-write reading bound the negation to a closed
list of acting verbs that missed the cards' own (lend, check out, hand,
pay out, ...), so "I didn't lend out AST-0041" read as no decline. The
list now carries every verb the cards' operation prompts ask for, and a
test enumerates each card operation's verb so a new card's verb fails it.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FEkQ5nw3MDkRymEzv6FDJs
…t in the verb enumeration

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FEkQ5nw3MDkRymEzv6FDJs
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface labels Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

Coarse fallback — 1 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 19cc8976bfc88e6314e0e9d41be012a1035279e9 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 8794c302e2a9400435b591e391602eaf677932bc — the merge of head f38ab869cd35de136d61225ef2771f2e906ff416 into base 19cc8976bfc88e6314e0e9d41be012a1035279e9, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 8794c302e2a9400435b591e391602eaf677932bc && git checkout 8794c302e2a9400435b591e391602eaf677932bc
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 19cc8976bfc88e6314e0e9d41be012a1035279e9 f38ab869cd35de136d61225ef2771f2e906ff416 && git checkout -B drift-repro 19cc8976bfc88e6314e0e9d41be012a1035279e9 && git merge --no-ff f38ab869cd35de136d61225ef2771f2e906ff416

node scripts/docs-audit/affected-docs.mjs --json 19cc8976bfc88e6314e0e9d41be012a1035279e9

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:devx#4 · os-project-manager · session_01FEkQ5ooRyyS95kdyC2dPEX

席位核对 · ACCEPT · head f38ab869 · 2026-11-14T21:55Z

  • 形态: draft → main, attribution first, Fixes #33415. Paths from the PR's file list: .changeset/33415-scenario-grader-card-verb-decline.md (+12/-0), packages/qa/scenarios/src/score/scenario-run-grade.ts (+10/-1), packages/qa/scenarios/test/runner/fixtures/asset-lifecycle-withheld-loan.recording.json (+24/-0), packages/qa/scenarios/test/runner/mid-tier-run-readings.test.ts (+101/-0), packages/qa/scenarios/test/score/scenario-run-grade.test.ts (+26/-0). No governed path, no packages/spec/src/**, Clause-②: no ⇒ no contract-review face; lane seat lands.
  • Diff read: Grader only, card Ask 1. WITHHELD_ACT gains the verbs the nine cards' operation prompts use (lend/lent, log, check … out with up to three words between, move, dispose, add, remove, fill, enter, tell, hand, resolve, close, file, pay/paid); a bare "I didn't check" stays an inspection report, not a decline. A new test enumerates all 51 card operations and fails on a verb the list misses; the AST-0041 reply now reads as a decline while the existing negation-of-a-finding negatives stay "no decline"; ablation (drop lend|lent) turns 3 pins red. Re-grade on a one-turn recorded-driver fixture (fail → pass). Ask 2 (wording-light) rejected in the body because a finding-negation naming the target would read as a decline.
  • 门禁 check-runs (head f38ab869): 49 runs · 33 success · 16 skipped · 0 not green; TypeScript Type Check success, Lint & Repo Gates success. K = 12 merges behind main (< 15).
  • Changeset: one patch changeset for @objectstack/scenarios (published), .changeset/33415-scenario-grader-card-verb-decline.md, file name unique beside Scenario grader: a controlled_by_parent detail narrowed by its own read policy fails row-level-security (rowSecurityGap follows the master only) #33414.
  • 接受的偏差: the re-grade uses a fixture, since the recorded run is not reachable.
  • 范围外去向: none. Skills-docs: not owed (Metadata-protocol: none).
  • 落地条款: non-governed, all checks green ⇒ the seat readies and arms auto-merge. Ruling-ref: the maintainer's standing authorisation 「按照 pm-dispatch 规则复核过的都授权」 (2026-10-25T11:35Z) and 「你应该使用 pm-dispatch 技能审核和落地pr」 (this account's project).

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review November 14, 2026 21:56
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 43645466732 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

分类:failure —— 按下面的日志分诊。

失败的 job(日志抽取,best effort):

  • Dogfood Regression Gate (3/3) — 失败步骤: Boot example apps and exercise real user flows

    @objectstack/dogfood:test:  FAIL   isolated  test/bare-name-one-holder-flow-family.dogfood.test.ts > one package holder per flow, webhook, connector and email template name, on a booted showcase > the
      ↳ 失败原因: @objectstack/dogfood:test: AssertionError: one warning line: expected [] to have a length of 1 but got +0
    @objectstack/dogfood:test:  FAIL   isolated  test/bare-name-one-holder-flow-family.dogfood.test.ts > one package holder per flow, webhook, connector and email template name, on a booted showcase > ins
      ↳ 失败原因: @objectstack/dogfood:test: AssertionError: {"success":true,"data":{"manifestId":"com.qa.bare.delta","version":"1.0.0","versionId":"1.0.0","installedAt":"2026-11-14T22:00:39.213Z","hotLoaded":true,"upg
    @objectstack/dogfood:test:  FAIL   isolated  test/bare-name-one-holder-flow-family.dogfood.test.ts > one package holder per flow, webhook, connector and email template name, on a booted showcase > the
      ↳ 失败原因: @objectstack/dogfood:test: AssertionError: expected [ Array(3) ] to deeply equal [ Array(2) ]
    @objectstack/dogfood:test:  FAIL   isolated  test/bare-name-one-holder-flow-family.dogfood.test.ts > one package holder per flow, webhook, connector and email template name, on a booted showcase > the
      ↳ 失败原因: @objectstack/dogfood:test: AssertionError: expected [ Array(3) ] to deeply equal [ Array(2) ]
    @objectstack/dogfood:test:  FAIL   isolated  test/bare-name-one-holder-flow-family.dogfood.test.ts > one package holder per flow, webhook, connector and email template name, on a booted showcase > the
      ↳ 失败原因: @objectstack/dogfood:test: AssertionError: expected [ [ 'com.qa.bare.alpha', true ], …(2) ] to deeply equal [ [ 'com.qa.bare.alpha', true ], …(1) ]
    @objectstack/dogfood:test:  FAIL   isolated  test/bare-name-one-holder-flow-family.dogfood.test.ts > one package holder per flow, webhook, connector and email template name, on a booted showcase > the
      ↳ 失败原因: @objectstack/dogfood:test: AssertionError: expected [ 'com.qa.bare.alpha', …(3) ] to deeply equal [ 'com.qa.bare.alpha', …(2) ]
    

↳ 失败原因 是判读的关键:超时(Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言(AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

⚠️ 断言这一侧有一类例外,判据是断言在测什么,不是它是不是 AssertionError。 断言的对象是产品行为(一个值、一个形状、一次拒收)⇒ 照上面读:真实的行为改变,去查,⛔ 不要重排掉;
断言的对象是这次实验自身的有效性前提(跑完的耗时、负载下的先后、任何只在时间预算内才成立的条件)⇒ 它跟超时是同一类,同样对负载敏感,重排一次是合法的判别手段。
识别是机械的:断言的消息或它比较的值本身点名了一段时长、一个时间戳、一个耗时计数。实测过的一对 —— AssertionError: SecurityPlugin.init() ran: expected false to be true 测的是产品行为(真回归);
AssertionError: this run took over a second, so second-precision stamps could have differed too: expected 1006 to be less than 1000 测的是实验前提:它守护的那条不变式当时是绿的,同一个 head 原样重排一次即成功。
穿着 AssertionError 外衣的时间测量,仍然是时间测量。(⛔ 这只改「怎么读一次红」,不改「哪些测试可以重排」——后者由别处管。)

跨 PR 相同签名(24h,按失败测试文件聚合):

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 12 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:devx#4 · os-project-manager · session_01FEkQ5ooRyyS95kdyC2dPEX

席位核对 · re-queue once · 2026-11-14T22:14Z

Merged via the queue into main with commit d7f76bf Nov 14, 2026
50 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/project-thread-rl158j branch November 14, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scenario grader: "I didn't lend out AST-0041" is not read as a decline (WITHHELD_ACT's closed verb list lacks the cards' domain verbs)

1 participant