[pietern]

Changes

Add a pr-review agent skill under .agents/skills/. It reviews a branch (or a given commit) against its merge base with main and prints inline comments, a review body, and the evidence and worksheet behind them. It never posts.

The skill works through fixed passes: correctness, duplicated mechanisms, fit with existing extension points, new error paths on absent input, sibling routes (create vs update, deploy vs destroy), user-facing names, stored state over time, doc comments, tests, and user-facing text. It reads the repo's .agents/rules/ for testing level and comment conventions.

Why

We want agent reviews to catch the kind of issues maintainers raise in this repo, not just generic bugs, and to know how often they do. The skill was written and tuned against past reviews on this repo rather than from a checklist.

Tests

Backtest against human reviews. Merged PRs, each frozen at the commit that received the most human review comments, were reviewed by the skill with only read-only git access (no PR description, no GitHub), and an LLM judge checked whether it raised the same issue as each substantive human comment (match 1, partial 0.5).

• 24 PRs, 96 substantive human comments, split into a tuning set (12 PRs) and a held-out set (12 PRs) whose comments were not read while writing the skill.
• On held-out PRs the skill reproduces 41% of substantive comments, with 5.4 comments per PR. An earlier version that asked for three to eight comments per review reproduced 34% (range 30–38% over two runs) with 5.1 comments per PR, and a first generic version 31%. The largest gains are on PRs with many human comments, where the comment budget used to cut real findings.
• Comments on tests and acceptance output are caught least often (about 28% vs 41% for source, measured on the earlier version).

Sweep of recently merged PRs. The skill reviewed the 59 human-authored PRs merged into main between 2026-10-27 and 2026-11-03, at their final commit. It made 200 findings (about 3.4 per PR); 34 had already been raised on the PR. Each finding was then checked against the code on its own by a separate model told to try to refute it; that checker refuted all five deliberately false claims planted to test it. 193 findings were confirmed, 6 refuted and 1 left inconclusive. Of the confirmed ones not raised on the PR, 12 were bugs still present on main. Fixing them showed 2 to be working as expected; the other 10 are draft PRs #7331–#7340, each naming the PR that introduced the issue. The refuted findings were mostly behavior that the code or a test already documented as intended.

The sweep was run with the earlier version that asked for three to eight comments per review.

Each review takes 2–5 minutes with Opus. The backtest harness and fixtures are kept outside this repo so future runs can't read the expected comments.

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 552a987

Run: 41841800906

	Env	🔄​flaky	✅​pass	🙈​skip	Time
🔄​	aws linux-2core-8gb	2	269	15	4:34
✅​	aws-windows-latest-4core-16gb		273	13	3:30
✅​	azure linux-2core-8gb		270	15	4:47
✅​	azure-windows-latest-4core-16gb		272	13	3:50
✅​	gcp linux-2core-8gb		271	15	4:50
✅​	gcp-windows-latest-4core-16gb		273	13	3:56

	Test Name	aws linux-2core-8gb
🔄​	TestFsMkdirCreatesIntermediateDirectories	🔄​f
🔄​	TestFsMkdirCreatesIntermediateDirectories/uc-volumes	🔄​f

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:25	aws-windows-latest-4core-16gb	TestAccept
3:22	gcp-windows-latest-4core-16gb	TestAccept
3:20	azure-windows-latest-4core-16gb	TestAccept