[aknight-ld]

What

Criterion ERROR events now report failures the way ingest reads them, and rows whose generation failed are no longer scored.

Behavior change

• Error fields. A criterion ERROR event carries a top-level errorMessage, which falls back to "Unknown error".
  • The nested error: {code, message} object is no longer sent. Its code was a string cause name that ingest ignores.
  • errorCode stays omitted until integer codes are assigned.
  • Cause names remain internal, and still appear as error.type on the evaluate span.
• Empty messages. An error with an empty message now sends "Unknown error". Before, it sent the repr of the internal error mapping.
• Rows that did not generate. A row whose generation did not complete gets no judge call, no scorer call and no criterion event. Its generation ERROR event already records the row as failed, so a criterion event for it was never accounted for. These rows no longer take a concurrency slot. The generation_incomplete cause is removed.

The events module isn't part of the public API report, so there is no API change.

Tests

• Error cases: judge and scorer errors (invalid_judge_output, scorer_raised, invalid_score) assert errorMessage and the absence of error and errorCode. They tell causes apart through the internal results.
• Fallback: a unit test covers the "Unknown error" fallback.
• Failed generation: a two-row run, where one row raises during generation, asserts the following:
  • judges and scorers run only for the generated row;
  • no criterion event is tracked for the failed row;
  • the run still flushes and polls.
• Mutation check: all 8 new or changed cases fail against the previous source.

make lint format-check typecheck api-report-check and uv run pytest pass: 3636 passed, 18 skipped.

🤖 Generated with Claude Code

---

Note

Overview
Criterion ERROR events now match what ingest expects: only a top-level errorMessage is sent (with "Unknown error" when the message is empty). The nested error object is dropped from CriterionEventPayload and tracking payloads; string cause codes stay on internal results and error.type on evaluate spans.

Failed generation rows are no longer scored. _run_criteria_for_results runs judges and scorers only for rows with status == "COMPLETE", so errored rows get no criterion events and no concurrency slots—the generation ERROR event alone accounts for the row. The generation_incomplete criterion path is removed from scorer/judge helpers and docs.

Tests assert wire shape (no error / errorCode), cover scorer/judge failures via internal results, and replace the old “generation incomplete criterion event” case with a two-row test that proves only the successful row is judged/scored.

^{Reviewed by Cursor Bugbot for commit cc12093. Bugbot is set up for automated code reviews on this repo. Configure here.}