You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Record when a model's reply was cut off, so a run can show it #170
When a reply hits its token limit, the row should say so. Today nothing in a stored row shows that a judge, the simulated user, the assistant or a scorer's model was cut off, and a cut-off reply is a common cause of a wrong verdict: a judge that stops before its <rating>, a user turn that ends mid-sentence, an assistant reply that ends early and is scored as complete.
#116 makes the reason visible to code: acall_llm returns each call's finish_reason ("length" for a cut-off reply, Anthropic's max_tokens mapped to it) and usage. This issue is about keeping it in the row.
Why can't you do it today?
A probe can read finish_reason in its own hooks after #116, but the loop's own calls (the user model, the gate, the inline judge, summaries) happen inside the loop, and nothing records their stop reason. SimulationOutcome keeps per-alias calls, tokens and wall clock; SimulationTrace records checks, not model calls.
What shape do you propose?
Recorded from _finish_call, where every call already reports its usage to the outcome builder. Options:
A per-alias count of cut-off replies, per_model_cut_offs, next to per_model_calls. Smallest; tells a report which model ran out of tokens and how often, not where.
A warning (WarningKind) with the turn and alias. Shows where; a row with a cut-off reply reads "completed with warnings".
A trace per cut-off call, with finish_reason in extra. Most detail; traces today are per check, so this would be the first per-call one.
We'd suggest 1, plus 2 for the loop's own judge and user calls, where a cut-off changes the conversation.
Results impact
Option 1 adds an outcome key. Option 2 can move rows from completed to completed-with-warnings.
+1, and I'd push one step further than recording it: a truncated reply shouldn't just be visible, it should be excludable from scoring.
The reason is visible in your own examples. A judge that stops before <rating> doesn't produce a wrong verdict on average — it produces a systematically biased one, because cut-off judges fail in a direction (no verdict → scored as compliant, or scored by a fallback that defaults to pass). Same for a simulated user turn that ends mid-sentence: the assistant's next reply is answering a truncated prompt, so the turn is invalid, not merely degraded. Averaging those into a score silently moves the number.
with finish_reason normalised to a small enum — complete | length | content_filter | error | unknown — and keep the provider's raw string beside it. Then:
Derive, don't duplicate.per_model_cut_offs (your option 1) is a cheap GROUP BY alias WHERE finish_reason='length' over rows you already have; compute it at report time rather than maintaining a second counter that can drift from the rows. If rows are the source of truth, the counter is a view.
Make it a warning and a row flag. The warning tells a human where; the flag lets the scorer skip. Option 2 without the flag still lets a report say "73% agreement" over a set that includes invalid turns.
Report the truncation rate next to every score. "Score: 4.1/5 · 12% of judge calls truncated · 4 turns excluded" is honest; "4.1/5" alone is not, and the reader has no way to find out.
Fail the run, don't warn, above a threshold. If more than a few percent of judge calls hit the limit, the config is wrong (limit too small for the judge's chain-of-thought) and the numbers from that run shouldn't be published at all. A warning that everyone learns to ignore is worse than an error.
One normalisation trap: Anthropic reports max_tokens, OpenAI reports length, and some gateways report stop plus a separate truncated boolean. Put that mapping in exactly one function (normalise_finish_reason) with a test per provider, because the moment two adapters disagree your per-model counts become uninterpretable — and you'll blame the model for what was a mapping bug.
Activity
agent-mcocdaa commentedon Nov 20, 2026
+1, and I'd push one step further than recording it: a truncated reply shouldn't just be visible, it should be excludable from scoring.
The reason is visible in your own examples. A judge that stops before
<rating>doesn't produce a wrong verdict on average — it produces a systematically biased one, because cut-off judges fail in a direction (no verdict → scored as compliant, or scored by a fallback that defaults to pass). Same for a simulated user turn that ends mid-sentence: the assistant's next reply is answering a truncated prompt, so the turn is invalid, not merely degraded. Averaging those into a score silently moves the number.Concretely, I'd store on each call row:
with
finish_reasonnormalised to a small enum —complete | length | content_filter | error | unknown— and keep the provider's raw string beside it. Then:per_model_cut_offs(your option 1) is a cheapGROUP BY alias WHERE finish_reason='length'over rows you already have; compute it at report time rather than maintaining a second counter that can drift from the rows. If rows are the source of truth, the counter is a view.One normalisation trap: Anthropic reports
max_tokens, OpenAI reportslength, and some gateways reportstopplus a separatetruncatedboolean. Put that mapping in exactly one function (normalise_finish_reason) with a test per provider, because the moment two adapters disagree your per-model counts become uninterpretable — and you'll blame the model for what was a mapping bug.