Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/group-dispatch-scenarios.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"github-agent-workflow": patch
---

The five group skills have scenarios (three each). A group scenario lists `expect.dispatches`, the group's `results` and its one run-log `entry`, and for `triage-github-issues` the `reconcile` call that comes first. The checker picks the batch from the real item skill's eligibility oracle, oldest first, so a dispatch of an ineligible item, one past the batch, or in the wrong order fails. It reads what a dispatch must carry from the real item contract, `write_log=false` and the run's inputs that the item lists in `params`, so a dispatch that drops `trusted_authors`, or forwards something the item does not list, fails. `explore-github-repo` is checked as its own kind of group: findings ranked best first, one call to `file-github-issues` with `limit` and the shared inputs, only filed issues counting toward the cap, and the count line and deferred list in the log. The coverage gate wants three scenarios for a group skill, one that dispatches and one that does not. See ADR 0012.
10 changes: 10 additions & 0 deletions docs/adr/0012-scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,13 @@ A **scenario** is a world, a skill and its inputs, and the expected outcome: one
- **The goldens can be wrong in the same way as the contract.** Both came from the same reading of the skills. A scenario that a real run contradicts should change one of them.
- **Scopes are checked, not the lists under them.** The contract's `effects.writes` entries are prose, so an effect is checked against its scope and not against the exact sentence.
- **An oracle duplicates the prose.** Each oracle is `requires` written a second time, so the two can drift. A test compares the number of `requires` entries with `ORACLE_REQUIRES` in `scripts/scenario.mjs` and fails when they differ. It is a prompt to look, not a proof: rewording an entry without changing the count goes unnoticed. Group skills have no oracle.

## Update: group skills

A group skill has scenarios too. What a group promises is a selection and a set of dispatches, so its scenario says that instead of a result and effects: `expect.dispatches` (the items in order, the inputs each was given and the result each returned), `expect.results` (what the group returns), `expect.log` (the one run-log entry) and, for `triage-github-issues`, `expect.reconcile` (the call that comes first). Running the dispatched items is not checked here: they have scenarios of their own.

- **The checker picks the batch itself.** It orders the world's items by `createdAt` (by number when they have none), keeps those that the item skill's own oracle calls eligible, and takes the first `*_batch`. A scenario that dispatches an ineligible item, one past the batch, a different order, or leaves an eligible one out, fails. An item that fails the oracle does not use up the batch.
- **What a dispatch must carry is read from the item's contract.** It is `write_log=false` and every input of this run that the item lists in `params`, and nothing else. A dispatch that drops `trusted_authors`, or forwards a batch size the item does not list, fails. An input the run did not set is not forwarded, so the item falls back to its default.
- **The log entry lists the dispatched results in order**, with what reconciling did for triage, or `nothing eligible` when both are empty. The idle entry is always written.
- **`explore-github-repo` is the odd one.** Its candidates are findings it ranks (`findings: [{title, score}]`), and it calls `file-github-issues` once, with every finding best first and `limit` set to its batch. The checker verifies the order, the forwarded inputs, that only `filed` and `would-file` count toward the limit and everything after it is `deferred: over limit`, the `<N> filed, <M> commented, <K> dropped` line, and the deferred list in the log (at most 20). Its contract has no `nothing eligible` line, so an empty round returns the zero count line and calls nothing.
- **Coverage** wants three scenarios for each group skill, one that dispatches and one that does not.
32 changes: 15 additions & 17 deletions scripts/scenario-coverage.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,14 @@
// The same numbers are enforced by tests/scenarios.test.mjs: a skill cannot declare a result line
// that no scenario exercises.
import { readFileSync, readdirSync, existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import { parse } from "yaml";
import { extractContract } from "./contracts.mjs";
import { PKG, skillDirs, contractOf } from "./skill-contracts.mjs";
import { matchesReturn, parseDocLine, docHead, ORACLES } from "./scenario.mjs";

const PKG = join(dirname(fileURLToPath(import.meta.url)), "..");
const SKILLS = join(PKG, "skills");
/** Every skill directory in the package, by name: `skills/<bucket>/<skill>`. */
const skillDirs = () => readdirSync(SKILLS, { withFileTypes: true }).filter((b) => b.isDirectory()).flatMap((b) =>
readdirSync(join(SKILLS, b.name), { withFileTypes: true })
.filter((d) => d.isDirectory() && existsSync(join(SKILLS, b.name, d.name, "SKILL.md")))
.map((d) => ({ name: d.name, dir: join(SKILLS, b.name, d.name) })));
/** The parsed contract of a skill, or null when it has none (yet) or is not a skill of this package. */
export function contractOf(name) {
const found = skillDirs().find((s) => s.name === name);
const block = found && extractContract(readFileSync(join(found.dir, "SKILL.md"), "utf8"));
return block ? parse(block.yaml) : null;
}
export { contractOf };

/**
* @returns {string[]} the scenarios of a skill with an eligibility oracle that completed a run (not `skipped`) and
* have no `rerun`: what a second run would find is not said, and so not checked. An interrupted scenario has one.
Expand Down Expand Up @@ -49,7 +38,16 @@ export function coverage() {
const rows = [];
for (const { name } of skillDirs().sort((a, b) => a.name.localeCompare(b.name))) {
const c = contractOf(name);
if (!c || c.kind === "group" || c.kind === "reference") continue; // a group's results are its items' results; reference material is never run
if (!c || c.kind === "reference") continue; // reference material is never run
if (c.kind === "group") { // what a group promises is its selection: a round that dispatches, and one with nothing to do
const list = scenariosOf(name);
const missing = [];
if (list.length < 3) missing.push(`at least 3 scenarios (has ${list.length})`);
if (!list.some((sc) => (sc.expect?.dispatches ?? []).length > 0)) missing.push("a scenario that dispatches");
if (!list.some((sc) => (sc.expect?.dispatches ?? []).length === 0)) missing.push("a scenario with nothing eligible (no dispatch)");
rows.push({ name, total: 3, missing });
continue;
}
if (c.kind === "utility") {
const first = parseDocLine(c.returns[0]);
const verdicts = first.alts.filter((a) => !a.includes("<")).map(docHead);
Expand Down
145 changes: 145 additions & 0 deletions scripts/scenario.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

import { isDeepStrictEqual } from "node:util";
import { readFileSync } from "node:fs";
import { contractOf } from "./skill-contracts.mjs";

export const MUTEX_ISSUE = ["needs-info", "ready-for-agent", "ready-for-human", "agent:pr-open"];
export const MUTEX_PR = ["agent:needs-work", "agent:ready-for-review", "agent:human-merge"];
Expand Down Expand Up @@ -561,6 +562,149 @@ export function checkPureScenario(contract, sc) {
return out;
}

// ---- group skills: which items are dispatched, in what order, and with which inputs ---------------------
// A group scenario has `inputs` (the run's), a `world`, and `expect`: `dispatches` ([{target, inputs, result}], in order),
// `results` (what the group returns), `log` ({results}, the one run-log entry) and, for a group that calls another skill
// first, `reconcile` ({inputs, results}). The batch is picked here, from the real item skill's oracle, and what must be
// forwarded is read from the real item skill's contract. Running the dispatched items is not checked: they have their own scenarios.

/** The inputs a dispatch must carry: `write_log=false`, and every input of this run that the callee lists in `params`. */
export function forwardedInputs(callee, runInputs = {}) {
return { write_log: false, ...Object.fromEntries(Object.entries(runInputs).filter(([k]) => (callee.params ?? []).includes(k))) };
}

/**
* `explore-github-repo` is a group skill whose candidates are findings, not items of the world: it ranks them
* (`findings: [{title, score}]`, score = impact x confidence), calls `file-github-issues` once with all of them,
* best first, and `limit` = its batch, and logs what was filed, commented, dropped and deferred.
* `expect.dispatches` is that one call: `{inputs, candidates, results}`.
*/
function checkFindingsScenario(contract, sc, callee, calleeName) {
const out = [];
const bad = (m) => out.push(m);
const E = sc.expect ?? {};
const inputs = sc.inputs ?? {};
const findings = sc.findings ?? [];
const ranked = [...findings].sort((a, b) => b.score - a.score).map((f) => f.title);
const batch = Number(inputs.explore_batch);
if (!Number.isInteger(batch) || batch < 1) bad("the scenario must set `explore_batch` to a whole number of at least 1");
const calls = E.dispatches ?? [];
if (calls.length > 1) bad(`calls ${calleeName} ${calls.length} times, and the contract has it called once`);
if (findings.length === 0 && calls.length) bad(`calls ${calleeName} with nothing found, so it invents work`);
if (findings.length && calls.length === 0) bad(`found ${findings.length} candidate(s) and never calls ${calleeName}`);
let counts = { filed: 0, commented: 0, dropped: 0 }, deferred = [];
const [call] = calls;
if (call) {
if (!isDeepStrictEqual(call.candidates, ranked)) bad(`passes the candidates ${JSON.stringify(call.candidates)}, but best first is ${JSON.stringify(ranked)}`);
const { run_id, ...given } = call.inputs ?? {};
if (typeof run_id !== "string" || run_id === "") bad(`the call to ${calleeName} drops \`run_id\``);
const want = { limit: batch, ...Object.fromEntries(Object.entries(inputs).filter(([k]) => (callee.params ?? []).includes(k))) };
for (const k of Object.keys(want)) {
if (!(k in given)) bad(`the call to ${calleeName} drops \`${k}\``);
else if (!isDeepStrictEqual(given[k], want[k])) bad(`the call to ${calleeName} gives ${k}=${JSON.stringify(given[k])}, not ${JSON.stringify(want[k])}`);
}
for (const k of Object.keys(given)) if (!(k in want)) bad(`the call to ${calleeName} gives \`${k}\`, which it does not list or this run did not set`);
const results = call.results ?? [];
if (results.length !== ranked.length) bad(`${results.length} result line(s) for ${ranked.length} candidate(s): one each, in input order`);
let used = 0;
for (const [i, r] of results.entries()) {
if (!matchesReturn(r, callee.returns)) { bad(`\`${r}\` is not one of ${calleeName}'s returns`); continue; }
const title = ranked[i];
if (/^filed #/.test(r)) { counts.filed++; used++; if (inputs.dry_run === true) bad("files an issue with dry_run=true"); }
else if (/^would-file:/.test(r)) { used++; if (inputs.dry_run !== true) bad("`would-file` without dry_run=true"); }
else if (/^duplicate of #\d+ \(commented\)$/.test(r)) counts.commented++;
else if (/^dropped:/.test(r)) counts.dropped++;
else if (r === "deferred: over limit") { deferred.push(title); if (used < batch) bad(`defers \`${title}\` before the limit of ${batch} is reached`); }
if (r !== "deferred: over limit" && deferred.length) bad(`gives \`${r}\` after deferring a candidate: everything after the limit is deferred`);
}
if (used > batch) bad(`${used} candidates filed (or would-file) with a limit of ${batch}`);
}
const line = `${counts.filed} filed, ${counts.commented} commented, ${counts.dropped} dropped`;
if (E.result !== line) bad(`returns \`${E.result}\`, but the results count \`${line}\``);
const logged = E.log?.deferred;
if (!Array.isArray(logged)) bad("no expected run-log entry (`log: {deferred: [...]}`)");
else if (!isDeepStrictEqual(logged, deferred.slice(0, 20))) bad(`the log defers ${JSON.stringify(logged)}, but ${JSON.stringify(deferred.slice(0, 20))} were deferred (at most 20)`);
if ((E.effects ?? []).length) bad("a group skill does no item work, so it has no effects of its own besides its log entry");
return out;
}

export function checkGroupScenario(contract, sc) {
const out = [];
const bad = (m) => out.push(m);
const E = sc.expect ?? {};
const world = sc.world ?? { issues: [], prs: [] };
const inputs = sc.inputs ?? {};
const itemName = contract.dispatches;
const item_ = contractOf(itemName);
if (!item_) { bad(`dispatches ${itemName}, which has no contract`); return out; }
const compare = (what, given, want, callee) => {
for (const k of Object.keys(want)) {
if (!(k in given)) bad(`${what} drops \`${k}\``);
else if (!isDeepStrictEqual(given[k], want[k])) bad(`${what} gives ${k}=${JSON.stringify(given[k])}, not ${JSON.stringify(want[k])}`);
}
for (const k of Object.keys(given)) if (!(k in want)) bad(`${what} gives \`${k}\`, which ${callee} does not list or this run did not set`);
};

// selection: the item skill's `requires`, oldest first, the first `batch` that pass
const kind = (item_.params ?? []).includes("pr") ? "pr" : "issue";
const batchKey = (contract.params ?? []).find((k) => k.endsWith("_batch"));
const batch = Number(inputs[batchKey]);
if (!Number.isInteger(batch) || batch < 1) bad(`the scenario must set \`${batchKey}\` to a whole number of at least 1`);
const list = (kind === "pr" ? world.prs : world.issues) ?? [];
const dated = list.every((i) => i.createdAt !== undefined);
const oldestFirst = [...list].sort((a, b) => (dated ? Date.parse(a.createdAt) - Date.parse(b.createdAt) : a.number - b.number));
const oracle = ORACLES[itemName];
if (!oracle) return checkFindingsScenario(contract, sc, item_, itemName); // candidates are findings, not items of the world
const fwd = forwardedInputs(item_, inputs);
const eligible = oldestFirst.map((i) => `${kind}#${i.number}`).filter((t) => oracle(world, t, fwd).eligible);
const want = Number.isInteger(batch) && batch >= 1 ? eligible.slice(0, batch) : [];
const dispatches = E.dispatches ?? [];
const got = dispatches.map((d) => d.target);
for (const t of got) {
if (!item(world, t)) bad(`dispatches ${t}, which is not in the world`);
else if (!eligible.includes(t)) bad(`dispatches ${t}, which is not eligible (${oracle(world, t, fwd).reason})`);
else if (!want.includes(t)) bad(`dispatches ${t}, which is past the batch of ${batch}`);
}
for (const t of want) if (!got.includes(t)) bad(`does not dispatch ${t}, which is eligible and within the batch of ${batch}`);
if (got.length === want.length && got.every((t) => want.includes(t)) && got.some((t, i) => t !== want[i])) bad(`dispatches in the wrong order: oldest first is ${want.join(", ")}`);

// forwarding and results
for (const d of dispatches) {
compare(`the dispatch of ${d.target}`, d.inputs ?? {}, fwd, itemName);
const n = String(d.target).match(/#(\d+)$/)?.[1];
if (typeof d.result !== "string" || !d.result.startsWith(`#${n} `) || !matchesReturn(d.result, item_.returns)) bad(`${d.target}'s result \`${d.result}\` is not one of ${itemName}'s returns`);
}
const itemResults = dispatches.map((d) => d.result);
const expectResults = itemResults.length ? itemResults : ["nothing eligible"];
if (!isDeepStrictEqual(E.results, expectResults)) bad(`returns ${JSON.stringify(E.results)}, but the dispatched items gave ${JSON.stringify(expectResults)}`);

// the skill the group calls first
const called = (contract.calls ?? [])[0];
let reconciled = [];
if (called) {
const callee = contractOf(called);
const r = E.reconcile;
if (!r) bad(`must call ${called} first, and the scenario has no \`reconcile\``);
else {
compare(`the call to ${called}`, r.inputs ?? {}, forwardedInputs(callee, inputs), called);
reconciled = r.results ?? [];
for (const l of reconciled) if (!matchesReturn(l, callee.returns)) bad(`\`${l}\` is not one of ${called}'s returns`);
}
} else if (E.reconcile) bad("this group skill calls no other skill, so it has no `reconcile`");

// the one run-log entry
const entry = E.log?.results;
if (!Array.isArray(entry)) bad("no expected run-log entry (`log: {results: [...]}`)");
else {
const all = [...reconciled, ...itemResults];
const wantLog = all.length ? all : ["nothing eligible"];
if (!isDeepStrictEqual([...entry].sort(), [...wantLog].sort())) bad(`the log entry lists ${JSON.stringify(entry)}, but ${JSON.stringify(wantLog)} happened`);
else if (!isDeepStrictEqual(entry.filter((l) => itemResults.includes(l)), itemResults)) bad("the log entry lists the dispatched results out of order");
}
if ((E.effects ?? []).length) bad("a group skill does no item work, so it has no effects of its own besides its log entry");
return out;
}

/**
* The same skill's eligibility oracle, asked about the same item in the world a run leaves behind: could a
* scheduler pick the item up again? A run that completes released its lease. A run that was `interrupted`
Expand All @@ -587,6 +731,7 @@ export function rerunEligibility(sc, after) {
*/
export function checkScenario(contract, sc) {
sc = expandScenario(sc);
if (contract.kind === "group") return checkGroupScenario(contract, sc);
if (contract.kind === "utility") return checkPureScenario(contract, sc);
const out = [];
const bad = (m) => out.push(m);
Expand Down
24 changes: 24 additions & 0 deletions scripts/skill-contracts.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// Finds a skill of this package by name, in any bucket, and parses its `## Contract`. Shared by the
// scenario checker (a group skill is checked against the contract of the item skill it dispatches) and
// the coverage report.
import { readFileSync, readdirSync, existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
import { parse } from "yaml";
import { extractContract } from "./contracts.mjs";

export const PKG = join(dirname(fileURLToPath(import.meta.url)), "..");
const SKILLS = join(PKG, "skills");

/** Every skill directory in the package: `skills/<bucket>/<skill>`. */
export const skillDirs = () => readdirSync(SKILLS, { withFileTypes: true }).filter((b) => b.isDirectory()).flatMap((b) =>
readdirSync(join(SKILLS, b.name), { withFileTypes: true })
.filter((d) => d.isDirectory() && existsSync(join(SKILLS, b.name, d.name, "SKILL.md")))
.map((d) => ({ name: d.name, dir: join(SKILLS, b.name, d.name) })));

/** The parsed contract of a skill, or null when it has none or is not a skill of this package. */
export function contractOf(name) {
const found = skillDirs().find((s) => s.name === name);
const block = found && extractContract(readFileSync(join(found.dir, "SKILL.md"), "utf8"));
return block ? parse(block.yaml) : null;
}
Loading
Loading