Eval scoring v2: attribute scenario-card losses to the right owner, then compare haiku / sonnet / opus #33595

epic:#33595#1 · os-tesla · session_01L9qiby1cctpVpRbJzjccvY

Filed by eval#1 · os-tesla · session_01L9qibzr7AespGQoymAnLtn.

Epic PM seat (concentrated exception)

Epic PM seat: 🟢 epic:#33595#1 · os-tesla · session_01L9qiby1cctpVpRbJzjccvY · seated 2026-11-15T02:22Z · Patrol: trig_01RaqSwCHtoceTYMDUtancKr 21 * * * * (hourly while M1 is in flight; PR and mailbox events wake it sooner) · Mailbox: PR #33602.

• Exception reason (references/seat-post-protocol.md 〈Epic 收尾与集中例外〉): the maintainer asked for an epic PM by name, verbatim 「立专题卡，然后派专题pm」 (2026-11-15, this project's eval seat thread). M1 is one chain owned end to end: Scenario grader: an unanswerable question is graded correct when the reply declines but also states a figure #33599's token rule depends on Scenario grader: the leak check reads a figure out of a record identifier (INV-003 is read as 3) #33598's (same function, gradePersonaRun), and the M1 close-out replay (eval#1) needs all four children landed under one grader.
• Reserved chain: Scenario grader: a 403 on an operation whose declared outcome is applied is charged to the operation, not to the build that denied it #33596 Scenario grader: Build does not check that each persona's permission set covers the reads and writes the card gives it #33597 Scenario grader: the leak check reads a figure out of a record identifier (INV-003 is read as 3) #33598 Scenario grader: an unanswerable question is graded correct when the reply declines but also states a figure #33599 (pm:epic). M2/M3 rows are not cards yet; they are filed on the maintainer's go after the M1 split summary, into the devx lane queue unless that go reserves them here.
• File territory: packages/qa/scenarios/src/score/**, packages/qa/scenarios/src/runner/{observe,orchestrate,scorecard}.ts, packages/qa/scenarios/src/schema.ts, their tests under packages/qa/scenarios/test/**, the scorer prose in docs/qa/scenarios/README.md and packages/qa/scenarios/README.md, and each child's own changeset. Why this wide: each child lands in a different file of the scorer, and other devx seats should stay off the scorer while M1 is in flight. No scenario card JSON (docs/qa/scenarios/*.json) changes in M1.
• Hot-file serial queue: packages/qa/scenarios/src/score/scenario-run-grade.ts → Scenario grader: the leak check reads a figure out of a record identifier (INV-003 is read as 3) #33598, then Scenario grader: an unanswerable question is graded correct when the reply declines but also states a figure #33599 (Scenario grader: an unanswerable question is graded correct when the reply declines but also states a figure #33599's thread opens after Scenario grader: the leak check reads a figure out of a record identifier (INV-003 is read as 3) #33598 merges). Scenario grader: a 403 on an operation whose declared outcome is applied is charged to the operation, not to the build that denied it #33596 and Scenario grader: Build does not check that each persona's permission set covers the reads and writes the card gives it #33597 run beside them on disjoint files.
• Concurrency: at most 3 card threads at once (cap 5 per seat).
• Hand-off at M1 close: Notify: landed to eval#1 (mailbox pm:mailbox — eval#1 · os-tesla #33469), which owns the milestone table and the close-out replay.

Filing-gate category: ③ maintainer-directed (verbatim, this project's eval seat thread, 2026-11-15): 「你建议的任务全部立卡，你负责开 devx 车道派发处理」 · 「立专题卡，然后派专题pm」. Also ④: the parent of the milestone children below.

Why (business)

• The eval lane measures whether an AI can build and then query an ObjectStack app correctly: customers building with Claude Code most likely use opus; the cloud product's default for both building and ask is haiku.
• Round 1 of the eval seat (3 control cards, scratch, published 18.4.0, 2026-11-14) lost 7 of 15 operation points, and 4 of them were the AI-built app's permission set refusing a write the persona should be allowed (403). The Build half scored full marks, so the score blamed the ask side for a build error.
• Until losses land on the right owner (build, ask, or grader), a haiku / sonnet / opus comparison and the per-release baseline measure the grader as much as the models. Without this the maintainer cannot answer "is haiku good enough as the cloud default".

Milestone table

Rolling split (references/seat-lifecycle.md 〈Epic 子树车道〉): only the current milestone is cards; the rest are rows.

Each milestone's close-out posts the split summary here (next cards, lanes, edges, cost); the next milestone's cards are filed on the maintainer's go.

Readers

• The epic PM (epic:#<this>#1, to be seated in its own project thread on the maintainer's 「派专题pm」) dispatches the M1 children to os-dev and lands them.
• eval#1 ([PM seat] eval — 🟢 os-tesla #33468) owns this table, replays recordings at each close-out, and writes the scorecards (internal only; no accuracy figure is published).

Dedup

gh api search/issues on objectstack, open and closed, 2026-11-15: is:issue "scenario grader" in:title → 4 (#33414 #33415 #33256 #30290, all closed and different defects); "scenario runner" in:title → 14 (all closed; #31160 added the model flags this epic uses); "scenario permission set" in:title → 0; injection_as_data → 0. No duplicate.