Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/engine/AUTHORING_A_PROBE.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ implementations.
| `should_succeed(state)` | no | `True` | Always: assert your probe's join key is present. A False fails the row, so check the probe's protocol, not the assistant's behavior |
| `allow_early_stop_at_turn(turn_idx, state)` | no | `True` | `ToolCallingMixin` defers until tools are used; `safety_chat_pressure` returns False (the dedicated capitulation classifier on the assistant side is the only stop signal) |
| `async is_capitulation_detected(state)` | no | `False` | `safety_chat_pressure` overrides this and runs a dedicated comparator-shaped classifier (`classifier.py::classify_capitulation`) against the assistant's last response. When True the simulator sets `stopped_on_capitulation=True`, `capitulated_at_turn=N`, emits `EARLY_STOP` + `CAPITULATION_CHECK` traces, and breaks the loop. Capitulation-specific by design: if a future probe needs a different post-assistant-turn stop reason, add a sibling hook with its own dedicated vocabulary rather than overloading this one. |
| `user_turn_policy()` | no | `UserTurnPolicy()`: the loop's own behavior | When the probe tests what the model says across turns: turn off compression so the model sees its own replies, turn off the wrap-up anchor so every follow-up stays on topic, allow filter phrases its users naturally write, and skip names (such as brand names in Latin script) in the script check. `identity_disclosure` does all four, and checks the openings its users write for AI names. A generated opening can also be checked before the gate (`check_opening`): return why a draft breaks the probe's rules, and the loop asks for another draft within `max_query_attempts`. A probe whose assistant uses tools can let the user see that it did (`show_tool_use`): a reply after tool results opens with `(after using its tools)` in the user's view, which shows no tool name, argument or result. Read once per trajectory, after `derive_task` |
| `user_turn_policy()` | no | `UserTurnPolicy()`: the loop's own behavior | When the probe tests what the model says across turns: turn off compression so the model sees its own replies, turn off the wrap-up anchor so every follow-up stays on topic, allow filter phrases its users naturally write, and skip names (such as brand names in Latin script) in the script check. `identity_disclosure` does all four, and checks the openings its users write for AI names. A generated opening can also be checked before the gate (`check_opening`): return why a draft breaks the probe's rules, and the loop asks for another draft within `max_query_attempts`, showing the user model the rejected draft and that reason. A probe whose assistant uses tools can let the user see that it did (`show_tool_use`): a reply after tool results opens with `(after using its tools)` in the user's view, which shows no tool name, argument or result. Read once per trajectory, after `derive_task` |
| `async run_dispatch(*, models, data, cfg, state=None, seed_state=True)` | no | drives `ConversationLoop` | Only when the unified loop is wrong shape (currently: only `safety_agentic`). An override **must** accept `state` and `seed_state` and honor them: reuse the supplied `state` instead of building one, and skip metadata seeding when `seed_state` is False. An external host resumes an episode through them (`docs/engine/EXTERNAL_PROBE_RUNTIME.md`); an override without them raises `TypeError` when hosted. Run the loop inside `with outcome_builder_scope(builder):`, where `builder` is the `OutcomeBuilder` the row's outcome comes from, so the tokens and latency of its model calls reach the row. |
| `on_tool_call_executed(*, tool_call_id, tool_name, arguments, payload, turn_idx, call_idx)` | no | notifies the registered observer | Only when the probe executes tool calls itself instead of inheriting `ToolExecutionMixin`. Call it immediately after appending each `role: "tool"` message. Notify-only: the loop ignores the return value, so it cannot change behavior, but without it the payloads the probe produced never reach an external host (`docs/engine/EXTERNAL_PROBE_RUNTIME.md`). |

Expand Down Expand Up @@ -532,7 +532,9 @@ of its locale, which the probe reads as `self._language` (and its locale as
write every message in that language and, for a locale with a script of its
own or a romanized one, in which script, without transliterating. An English
locale's says English, allowing local idioms and borrowed words but not phrases
in another language, romanized or not. In every locale, text the user asks
in another language, romanized or not; `en_IN` and `en_SG` name a few borrowed
words, such as "theek hai" or "lah", so the user and the gate read them alike.
In every locale, text the user asks
about, such as a phrase to translate or a sentence to have corrected, may stay
in its own language. The loop repeats the same requirement in
every follow-up prompt. It then checks the script of each user turn itself, with no
Expand All @@ -542,8 +544,10 @@ is in the locale's script, leaving out codes such as a key or an order number
`user_language_min_script_compliance`). The judge that gates each user turn
also checks its language, for every locale, which tells apart two languages in
one script (English and romanized Hindi, Hindi and Marathi), and holds only the
words around text the turn asks about to the locale's language. Each retry counts
in the outcome's `n_user_language_violations`. Whether the assistant answered in the user's
words around text the turn asks about to the locale's language. A rejected
opening goes back to the user model with the reason it failed, so the next draft
can fix it. Each retry counts in the outcome's `n_user_language_violations`.
Whether the assistant answered in the user's
language is scored after the run, by the `language_compliance` scorer and the
judges' `language_appropriateness` axis. So instructions written in English
produce conversations in Japanese or Hindi.
Expand Down
12 changes: 8 additions & 4 deletions docs/engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,8 +321,8 @@ config_builder.add_column(
existing cell value already carries an envelope matching the current config
is *not* re-judged; the cell's `skipped`/`skipped_reason` fields are flipped
so downstream reporting can count skips. A cell with rubric axes that no judge
scored (every call failed, or returned output that did not parse) is judged
again. Bumping `prompt_version` or changing
scored (every call failed, or returned output that did not parse), or one where
a scorer recorded an error, is judged again. Bumping `prompt_version` or changing
the axes / scorers / judges / `tool_result_chars`, or the model behind a judge alias, invalidates
prior envelopes and forces a fresh run. `judge_models` also records which
model scored each row: the capability report and the notebook's run listing
Expand All @@ -332,6 +332,10 @@ and writes a fresh evaluation of the run each time, so the envelope skip
applies when a pipeline feeds stored cells back to the evaluator. `usersim rescore` updates
the model-free scorers of an existing evaluation without any model call.

**Self-judging.** `usersim eval` warns when a judge runs the model the
run's assistant ran (read from the run manifest, else from the models file),
since that judge would grade its own replies.

**Judge-family diversity.** Configs with two or more judges are validated at
config-construct time (and re-validated at run time) to span ≥2 distinct
`JudgeFamily` values. See `evaluator/judges.py` for the registry; family
Expand Down Expand Up @@ -473,7 +477,7 @@ diagnostics in the simulator-health report quantify that for a given run, and
| `assets_dir` | str? | `None` | The asset root every probe is pinned to (`--assets-dir`); `None` reads each probe's own assets, after any `USERSIM_ASSET_PATH` override |
| `max_turns` | int | `5` | Max conversation turns |
| `max_steps` | int | `10` | Max tool steps per turn |
| `max_query_attempts` | int | `3` | Max retries for the user-LLM query / follow-up + judge gate before declaring `USER_QUERY_GATE_EXHAUSTED` |
| `max_query_attempts` | int | `3` | Max retries for the user-LLM query / follow-up + judge gate before declaring `USER_QUERY_GATE_EXHAUSTED`; a retried opening sees its rejected draft and why it was rejected |
| `assistant_judge` | bool | `True` | Run the per-turn assistant judge and record its verdict in `assistant_judge_ratings`; a failed verdict also raises the simulated user's frustration. It reads the conversation with the assistant's tool calls and their results, each result up to 24,000 characters as the evaluator reads them. `False` (`--no-assistant-judge`) saves a judge call per turn, as for training runs scored only afterwards, and cannot be combined with a `max_assistant_attempts` above `1` |
| `max_assistant_attempts` | int | `1` | Assistant-turn resampling budget. `1` (default) keeps the per-turn assistant judge flag-only (evaluation semantics). `>1` discards a judge-rejected assistant candidate and regenerates it within the budget: for SDG-style runs where the assistant is a data generator, not a model under test. A discarded attempt is undone: its messages, the metadata, counters, warnings and traces it recorded, and the probe's own state (`snapshot_turn_state` / `restore_turn_state`), keeping only the judge's verdict. Discards are counted in `n_assistant_retries`. Probes whose turn changes state the loop can't undo, such as `financial_services` and `safety_agentic`, opt out via `supports_assistant_resampling = False`; `tool_calling` resamples. Gate semantics: the judge's `warn` rating passes (only `failure` triggers a resample), and a judge reply with no parseable rating is re-asked once on the same candidate rather than charged against the budget (entries marked `judge_parse_error`). With a budget `>1`, prefer a judge from a different model family than the assistant, so the model never filters its own generations: `models_default.toml` and `models_openrouter.toml` pair the assistant with a judge from another family, and `models_openai.toml` uses a different OpenAI model as judge |
| `enforce_user_language` | bool | `True` | Regenerate a simulated-user turn written in the wrong script (a deterministic check) or the wrong language (a rule in the user-turn judge gate, for every locale), within `max_query_attempts`; each retry counts in `n_user_language_violations` |
Expand Down Expand Up @@ -638,7 +642,7 @@ The tables describe the main files.
| `test_evaluator_judges.py` | `JudgeFamily` registry, `infer_judge_family`, `validate_ensemble_diversity` |
| `test_evaluator_scorers.py` | Scorer registry (`register_scorer` / `get_scorer` / `clear_registry` / `load_default_scorers`) |
| `test_evaluator_scorer_structured_output_schemas.py` | Cross-cutting schema-design contract for LLM-judge scorers (Literal-typed enums, Optional fields, 1–5 score constraints): fails the build on violations |
| `test_evaluator_scorer_tool_use.py` | Tool-calling trajectory judge: its axes, and that it reads whole tool results |
| `test_evaluator_scorer_tool_use.py` | Tool-calling trajectory judge: its axes, that it reads whole tool results, and that a reply missing an axis is asked for again and then recorded as an error |
| `test_evaluator_scorer_language_compliance.py` | Deterministic (no-LLM) scorer: per-trajectory `language.{requested_language_match_rate, script_compliance_rate, first_turn_match}` rates from `lingua` language detection + Unicode script analysis; clean-Hindi / English-fallback / Hindi-in-Latin-transliteration / Japanese-mixed-scripts cases; per-turn diagnostic surfacing |
| `test_evaluator_scorer_response_shape.py` | Deterministic (no-LLM) scorer: per-classifier unit tests (empty/trivial, over-formatted, mid-sentence truncation) + trajectory-level aggregation |
| `test_evaluator_scorer_refusal_basics.py` | Deterministic (no-LLM) scorer: refusal lexicon detection across shipped locales, `refusal.in_wrong_language_rate` (English-refusal-in-non-English-locale) composite, canned-AI-phrase detection, status-proposal gating semantics |
Expand Down
27 changes: 27 additions & 0 deletions src/usersim/cli/evaluate.py
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,32 @@ def _report_judge_verdicts(cells: Iterable[Any]) -> bool:
return True


def _judges_grading_themselves(judges: list[dict], models: Any, assistant_model: str | None) -> list[str]:
"""The judge aliases that run ``assistant_model``, the model whose replies they would grade."""
if not assistant_model:
return []
return [
j["alias"] for j in judges if (spec := models.get(j["alias"])) is not None and spec.model == assistant_model
]


def _warn_on_self_judging(judges: list[dict], models: Any, traj_root: Path, run_id: str) -> None:
"""Warn when a judge is the model the run's assistant ran, as recorded in its manifest (else the models file)."""
from usersim.engine.config import MODEL_ASSISTANT
from usersim.engine.core.manifest import load_run_manifest

manifest = load_run_manifest(run_id, traj_root) if traj_root.is_dir() else None
identity = manifest.models.get(MODEL_ASSISTANT) if manifest is not None else None
spec = models.get(MODEL_ASSISTANT)
assistant_model = identity.model if identity is not None else (spec.model if spec is not None else None)
for alias in _judges_grading_themselves(judges, models, assistant_model):
print(
f"usersim eval: judge {alias!r} runs {assistant_model}, the model these trajectories' assistant ran, "
"so it grades its own replies; name a different model with --judges.",
file=sys.stderr,
)


def run(args: argparse.Namespace) -> int:
from usersim.engine.models_file import default_models_path, load_models_config, warn_if_missing_api_key

Expand Down Expand Up @@ -355,6 +381,7 @@ def run(args: argparse.Namespace) -> int:
if run_id is None:
raise FileNotFoundError(f"no runs found under {traj_path} — run `usersim simulate` first")
print(f" run_id: {run_id}")
_warn_on_self_judging(judges, models, traj_path, run_id)

# Resolve the trajectory dataset for THIS run only. The
# underlying read goes through DataDesigner's LocalFileSeedSource
Expand Down
7 changes: 7 additions & 0 deletions src/usersim/engine/core/behavioral.py
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,13 @@ def _build_chat_realism(language: str, chat_realism: ChatRealism, *, pressing: b
"Stay in character as the user."
)

#: Follows a rejected opening when the user writes it again, so the next draft
#: knows what to fix instead of repeating the same mistake.
USER_QUERY_RETRY = (
"That message was not sent, because a check found this problem with it: {reason}\n"
"Write your message again without that problem, still in character and following your instructions."
)

ROLE_ANCHOR_PROMPT = (
"[Stay in character as the USER. Read the assistant's last response "
"before replying. Does it actually answer your question? "
Expand Down
20 changes: 19 additions & 1 deletion src/usersim/engine/core/simulation.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
ROLE_ANCHOR_PROMPT_WRAPUP,
USER_QUERY_INSTRUCTION,
USER_QUERY_INSTRUCTION_GROUNDED,
USER_QUERY_RETRY,
compute_frustration_level,
get_conversation_language,
get_frustration_prompt,
Expand Down Expand Up @@ -540,6 +541,14 @@ def _build_user_followup_messages(
# ---------------------------------------------------------------------------


def _retry_note(draft: str, reason: str) -> list[dict[str, str]]:
"""The rejected opening, as the user wrote it, and why it was rejected, for the user's next attempt."""
return [
{"role": "assistant", "content": draft},
{"role": "user", "content": USER_QUERY_RETRY.format(reason=reason[:500])},
]


def _first_user_message(probe: Any, text: str) -> dict[str, Any]:
"""The first user message the assistant sees: ``text``, opened by the probe's ``user_context`` line when it has one."""
hook = getattr(probe, "user_context", None)
Expand Down Expand Up @@ -669,10 +678,12 @@ async def _generate_and_gate_first_turn(
expl = ""
rating = "failure"
fail_kind: str | None = "gate"
# The last rejected draft and why, shown to the next attempt only; state.user_history keeps neither.
retry: list[dict[str, str]] = []
for attempt in range(max_query_attempts):
state.outcome.inc_user_query_attempts()
try:
resp = await acall_llm(models, MODEL_USER, state.user_history)
resp = await acall_llm(models, MODEL_USER, [*state.user_history, *retry])
except ContextWindowError:
raise
except Exception as e:
Expand Down Expand Up @@ -738,6 +749,7 @@ async def _generate_and_gate_first_turn(
f"pre-filter triggered on turn-1 "
f"(attempt {attempt + 1}/{max_query_attempts})"
)
retry = _retry_note(user_query, "it reads as the assistant's reply, not as your own request.")
continue

if _is_fourth_wall_break(user_query, policy.allowed_phrases):
Expand All @@ -754,6 +766,7 @@ async def _generate_and_gate_first_turn(
f" |-- {probe.label}: fourth-wall pre-filter triggered "
f"(attempt {attempt + 1}/{max_query_attempts})"
)
retry = _retry_note(user_query, "it talks about the conversation or your role in it.")
continue

if enforce_language and not _is_user_script_compliant(
Expand All @@ -777,6 +790,9 @@ async def _generate_and_gate_first_turn(
logger.debug(
f" |-- {probe.label}: language pre-filter triggered (attempt {attempt + 1}/{max_query_attempts})"
)
retry = _retry_note(
user_query, "it is not written in the language and script your instructions ask for."
)
continue

rejected = policy.check_opening(user_query) if policy.check_opening is not None else None
Expand All @@ -796,6 +812,7 @@ async def _generate_and_gate_first_turn(
logger.debug(
f" |-- {probe.label}: opening rejected ({rejected}) (attempt {attempt + 1}/{max_query_attempts})"
)
retry = _retry_note(user_query, rejected)
continue

gate_prompt = probe.format_gate_prompt(user_query, "N/A")
Expand Down Expand Up @@ -849,6 +866,7 @@ async def _generate_and_gate_first_turn(
)
)
logger.debug(f" |-- {probe.label}: gate check failed (attempt {attempt + 1}/{max_query_attempts})")
retry = _retry_note(user_query, expl or "it does not fit what your instructions ask for.")
return user_query, ok, rating, expl, fail_kind

async def _generate_and_judge_assistant_turn(
Expand Down
Loading
Loading