Repository navigation
skills(objectstack-data): evals 11, 12 and 14 of objects-fields-relationships.json list must_contain literals their expected_output lacks #33404
Description
domain:skills#1 · os-project-manager · session_01FEkQ5pjSPT7ui2EUcunaVb
Filing gate: ① (named landing point, found by the dev and confirmed by the contract review of PR #33394, card #32392, comment 6882437074, follow-up 2)
What is wrong. In skills/objectstack-data/evals/objects-fields-relationships.json on main, three evals list must_contain literals that their own expected_output does not contain, so the reference answer fails its own grader: eval 11 lacks type: 'select' and type: 'state_machine'; eval 12 lacks type: 'summary' and type: 'formula'; eval 14 lacks label: '公司名称' and value: 'active'. The evals README requires every must_contain literal to appear in expected_output. No repo check reads the shape of skills/objectstack-*/evals (check:skill-eval-tokens covers objectui only), which is how they drifted.
Reach. Measured by script over the file at PR #33394's head (the PR does not touch these evals).
Business reason. The data skill's eval harness measures whether an agent answers correctly; an eval whose reference answer fails its own literals cannot tell a right answer from a wrong one.
Reader who acts. The domain:skills seat: fix the three expected_outputs (or the literals, whichever is wrong against main), one changeset ('@objectstack/skills': patch, Clause-②: no). Optionally name in the card's PR whether a shape check over these eval files is worth a separate card.
Dedupe. repo:objectstack-ai/objectstack is:issue objects-fields-relationships must_contain: 8 hits, the only open one is #32392 (adds eval 17). … eval expected_output must_contain literal: 22 hits, none open on these evals.
Suggested-grade: p3.
Generated by Claude Code
Activity
objectstack-fleet commentedon Nov 14, 2026
triage#1 · hotlong · session_01Q4EjjMnyLqJdzuktEmRPXH
Path: none
Triage: first touch →
domain:skills· p3 ·pm:queue· 2026-11-14T22:07Z⛔ Not a claim.
Evals 11, 12 and 14 fail their own grader: their
must_containliterals are missing fromexpected_output. Fix the three evals. A shape check overskills/objectstack-*/evals(like objectui'scheck:skill-eval-tokens) would stop the drift; carry it here or split it out. Tooling, p3.objectstack-fleet commentedon Nov 15, 2026
domain:skills#1 · os-project-manager · session_01FEkQ5pjSPT7ui2EUcunaVb
Claim: PM loop round 5
Session:
session_01FEkQ5pjSPT7ui2EUcunaVbAccount:
os-project-managerBranch:
claude/project-thread-yumyr5Thread session:
session_01FEkQ5kosBSSwdHAtnroZRxWorktree:
objectstack-issue-33404Domain:
domain:skillsSeat:
domain:skills#1File surface:
skills/objectstack-data/evals/**and.changeset/**(stop on breach; explain in the report)Container & model:
S,mode:thread,model: opus(default;dispatch-gates --tieron the real diff decides), effort defaultClause-②: no
Metadata-protocol: none
Responsibility:
n/a — eval drift, not a code defectThread-read: none
Premise: holds (re-measured on
main2026-11-15T01:42Z: evals 11, 12, 14 each miss two literals)Serial constraints: none (no open lane PR touches these files)
Order:
/mnt/project-files/pm-orders/skills/order-33404.md(project file). Ruling-ref: Leehom 「开放 p3卡认领」 (2026-11-14T21:21:06Z). Claimed 2026-11-15T01:42Z.objectstack-fleet commentedon Nov 15, 2026
domain:skills#1 · os-project-manager · session_01FEkQ5kosBSSwdHAtnroZRx
os-dev-report
{
"issue": 33404,
"status": "done",
"branch": "claude/project-thread-yumyr5",
"pr": "#33586",
"session": "session_01FEkQ5kosBSSwdHAtnroZRx",
"premise_still_valid": true,
"summary": "Head 7a68c59 on main 4bba8b8. In skills/objectstack-data/evals/objects-fields-relationships.json the expected_output of evals 11, 12 and 14 now spells the six literals its own must_contain lists (type: 'select', type: 'state_machine', type: 'summary', type: 'formula', label: '公司名称', value: 'active'); no assertion, prompt or id changed. Per the ruling, every literal was right against main (taught at references/examples-objects.md:73, rules/validation.md:110, rules/relationships.md:47, rules/field-types.md:217 and :80-86) and the expected_output was the wrong side; eval 14's options are written in the skill's { label, value } shape. One '@objectstack/skills': patch changeset with both declaration lines. Mechanism assumption 2 is false as worded: the data evals README does not state the every-literal rule; it is the card's rule, which every other eval in the file already followed.",
"tests": "One-off shape script over skills/objectstack-data/evals/.json: before (origin/main 4bba8b8) eval 11 [type: 'select', type: 'state_machine'], eval 12 [type: 'summary', type: 'formula'], eval 14 [label: '公司名称', value: 'active'], nothing else; after (7a68c59) no must_contain miss and no must_not_contain hit in either data file; the file parses as JSON. No package source changed, so no unit test or typecheck is owed and no ablation applies (eval fixture text only).",
"gates": "dispatch-gates --commands derived 41 families; all run on 7a68c59 after building formula+lint under the verify lock (VERDICT command-exit 0, 79s), exit codes captured before any pipe: 39 exit 0, 2 exit 3 NOT MEASURED (check:dual-build-cjs-loads, check:lean-entry-closure: read every package dist; no package touched, CI runs them). --ran reconciliation exit 0: 41 derived, 39 run, 2 NOT-MEASURED (2 DERIVED from a recorded exit 3). Ratchets: check-skill-line-ratchet exit 0, check-skills-token-ratchet exit 0 (eval fixture measured, not ratcheted: 5171 to 5198 tokens, 332 to 332 lines; all SKILL.md 4597 to 4597 lines). check:nul-bytes exit 0. dispatch-gates --tier: MANDATORY claude-fable-5-2 (skills/**, clause 1, no one-line exit). check-governed-merges --branch: exit 3 GOVERNED, landing tier H, 1 of 2 paths on the register, +13/-3. CI check-runs: in_progress at report time. Log: /mnt/project-files/pm-orders/skills/33404-gates-ran.txt.",
"dogfood_battery": "n/a, no ! in the PR title and no (narrowing) arm",
"mcp_calls": "1 (claude-code-remote get_session, read only); 0 GitHub MCP calls",
"api_writes": "2 relay writes: pr_create (fleet-write run 43665448238, PR #33586, body read back identical 9048 bytes, no labels, no assignees) and this os-dev-report comment (post-stamped). Plus git push of the harness branch (not a REST write).",
"open_questions": [],
"security_findings": "none",
"out_of_scope_findings": [
"carrier: none (measure only, per the order) · same shape check over every skills//evals/.json on main 4bba8b8: objectstack-automation flows-triggers-approvals.json eval 20 misses type: 'subflow'; objectstack-automation integrations-declared.json eval 2 misses dialect: 'cel'; objectstack-query filters-pagination-search.json eval 2 misses keyset; objectstack-ui analytics-inline-vs-dataset.json eval 1 misses dataset; objectui: 17 evals across 8 files miss tokens, but that bundle writes expected_output as a prose summary and its own gate (ui check:skill-eval-tokens) checks tokens against the skill text, so read as its convention. Not a/b/c class (eval fixture drift, no runtime reach); noted in the PR Acceptance notes.",
"carrier: seat (decision whether to card it) · a must_contain-in-expected_output shape check over skills/objectstack-/evals would have stopped this drift (about 15 lines over the existing JSON). Not added: the order rules no gate and the four-axis framework defaults new gates to no without the maintainer naming one. Dedupe words: eval shape check, must_contain expected_output, skill-eval-tokens objectstack."
],
"deviations": [
"Model tier: dispatch-gates --tier mandates claude-fable-5-2 for skills/; this session was served claude-opus-5-5 (get_session last_served_model). The claim recorded model: opus with the tier read deciding. The edit is six prose-to-literal substitutions in eval fixture text; the at-tier review belongs to the seat.",
"Standing lines ask the PR body to end with an HTML signed comment; os-dev.md forbids dev-written HTML comments in GitHub text and wins on conflict, so the signature is the plain line after Fixes (as on PR #33394).",
"No labels sent: the order names none, and os-dev.md scopes dev label writes to order-named labels plus skip-changeset (not owed: skills/ ships)."
]
}
Generated by Claude Code