Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/engine/AUTHORING_A_PROBE.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ implementations.
| `get_tools_for_assistant()` | no | `None` | When you expose tools (tool_calling, agentic) |
| `format_gate_prompt(user_query, history)` | no | generic rubric | When the gate needs probe-specific quality criteria |
| `format_assistant_judge_prompt(response, history)` | no | generic assistant-quality rubric | When the per-turn assistant judge needs probe-specific criteria. The loop calls it on every assistant turn unless the run switches the judge off (`assistant_judge=False`); with `max_assistant_attempts > 1`, its verdict also gates resampling. `history` holds the user and assistant text and the assistant's tool calls with their results, each result up to 24,000 characters as the evaluator reads them |
| `async after_assistant_turn(models, state, response, cfg)` | no | append + return content | When you intercept tool calls, simulate API responses, etc. Build the message with `assistant_message(response, content, tool_calls=..., store_reasoning=getattr(cfg, "store_reasoning", True))` from `usersim.extension` rather than a hand-written dict: it carries the model's `reasoning_content` (when there is one) onto the message it belongs to. A hand-built dict silently drops the trace, and for multi-call turns attributes it to the wrong message; omitting `store_reasoning` makes your probe ignore `--no-store-reasoning`. |
| `async after_assistant_turn(models, state, response, cfg)` | no | append + return content | When you intercept tool calls, simulate API responses, etc. Build the message with `assistant_message(response, content, tool_calls=..., store_reasoning=getattr(cfg, "store_reasoning", True))` from `usersim.extension` rather than a hand-written dict: it carries the model's `reasoning_content` (when there is one) onto the message it belongs to. A hand-built dict silently drops the trace, and for multi-call turns attributes it to the wrong message; omitting `store_reasoning` makes your probe ignore `--no-store-reasoning`. `response` also carries the call's `finish_reason` (`"length"` when the reply was cut off) and `usage` (its token counts), which `assistant_message` leaves out of the stored message. A test that patches `acall_llm` may not set them, so read them with `response.get(...)`. |
| `supports_assistant_resampling` (class attr) | no | `True` | Set `False` when a turn changes state the loop can't undo, such as a mock environment: it caps judge-gated assistant resampling (`max_assistant_attempts > 1`) to one attempt. The loop undoes a rejected attempt's messages, `state.metadata` and outcome record itself. The tool mixins set it `False`; a probe on them turns it back on when its hooks below cover the rest, as `tool_calling` does |
| `snapshot_turn_state()` / `restore_turn_state(snapshot)` | no | keep nothing | When the probe keeps state of its own that a turn can change and it supports resampling: return a copy of that state, and put it back. Override both or neither; `assert_probe_conforms` checks |
| `build_result_extras(state)` | no | num_turns + num_tool_calls | Always: surface the probe's join keys as top-level columns. **List every key it adds beyond the core columns in the class's `output_columns`** (see step 6) |
Expand Down
8 changes: 8 additions & 0 deletions docs/engine/EXTERNAL_PROBE_RUNTIME.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,11 +239,19 @@ while isinstance(event, ActivationRequest):
"tool_calls": reply.tool_calls,
},
"usage": {"input_tokens": reply.input_tokens, "output_tokens": reply.output_tokens},
"finish_reason": reply.finish_reason,
})

row = await runtime.finalize()
```

`usage` and `finish_reason` are optional; pass them when your provider reports
them. `finish_reason` is the provider's, as on an OpenAI choice (`"length"` for
a cut-off reply); Anthropic's stop reasons are mapped to OpenAI's as for a
configured model, so `"max_tokens"` also reads `"length"`. Probes see both
exactly as they would from a configured model, and neither is written into the
transcript.

Each `ActivationRequest` carries:

- `role`: `user`, `assistant`, `judge` or `summary`;
Expand Down
2 changes: 1 addition & 1 deletion docs/extension_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ does not load the conversation loop.
| `MODEL_ASSISTANT` | The alias of the assistant under test |
| `MODEL_JUDGE` | The alias of the model that checks user turns and judges assistant turns during the conversation, and checks tool_calling's simulated tool responses when the models file has no `api_response_judge_model` |
| `ConversationSimulatorConfig` | The run's settings, which hooks receive as `cfg` |
| `acall_llm` | Calls the model behind an alias and returns its reply as a message |
| `acall_llm` | Calls the model behind an alias and returns its reply as a message, with two keys that describe the call: `finish_reason`, why the model stopped in OpenAI's terms (`"length"` for a cut-off reply, None when the provider gave none), and `usage`, `{"input_tokens", "output_tokens"}` or None when the provider reported no counts. Only the real `acall_llm` returns them, so read them with `.get`; `assistant_message` leaves them out of the message it builds |
| `run_inline_judge` | Asks a judge alias for an `<explanation>` and a `<rating>`, reformatting once if needed; returns the explanation, the rating and whether it passed (`success` or `warn`) |
| `run_inline_judge_ex` | `run_inline_judge` plus whether the reply could be read: `False` when even the reformatted reply has no valid `<rating>`, so a caller can tell an unreadable verdict from a rejection |
| `resolve_model_name` | The model id behind an alias, or the alias when its model does not expose one |
Expand Down
14 changes: 12 additions & 2 deletions src/usersim/engine/core/episode_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,16 +163,21 @@ class ActivationResult:

``response`` is an OpenAI-shaped assistant message. Reasoning belongs in
``reasoning_content`` and is retained exactly as a configured model's would
be; ``usage`` feeds UserSim's per-model token accounting.
be; ``usage`` feeds UserSim's per-model token accounting. ``finish_reason``
is the provider's, as on an OpenAI choice (``"length"`` for a cut-off
reply); UserSim reports it to the caller exactly as for a configured model.
"""

activation_id: str
response: dict[str, Any]
usage: ActivationUsage | None = None
finish_reason: str | None = None

def __post_init__(self) -> None:
if not isinstance(self.activation_id, str) or not self.activation_id:
raise ValueError("activation_id must be a non-empty string")
if self.finish_reason is not None and not isinstance(self.finish_reason, str):
raise TypeError("activation finish_reason must be a string or None")
if not isinstance(self.response, Mapping):
raise TypeError("activation response must be a mapping")
object.__setattr__(self, "response", _normalize_assistant_response(self.response))
Expand All @@ -191,7 +196,9 @@ def from_value(cls, value: ActivationResult | Mapping[str, Any]) -> ActivationRe
usage = None
if isinstance(raw_usage, ActivationUsage):
usage = raw_usage
elif isinstance(raw_usage, Mapping):
elif isinstance(raw_usage, Mapping) and any(
raw_usage.get(name) is not None for name in ("input_tokens", "output_tokens")
):
usage = ActivationUsage(
input_tokens=int(raw_usage.get("input_tokens") or 0),
output_tokens=int(raw_usage.get("output_tokens") or 0),
Expand All @@ -200,6 +207,7 @@ def from_value(cls, value: ActivationResult | Mapping[str, Any]) -> ActivationRe
activation_id=str(value.get("activation_id") or ""),
response=deepcopy(dict(response)),
usage=usage,
finish_reason=value.get("finish_reason"),
)

def to_dict(self) -> dict[str, Any]:
Expand All @@ -208,6 +216,7 @@ def to_dict(self) -> dict[str, Any]:
"activation_id": self.activation_id,
"response": deepcopy(self.response),
"usage": self.usage.to_dict() if self.usage else None,
"finish_reason": self.finish_reason,
}


Expand Down Expand Up @@ -693,6 +702,7 @@ async def acompletion(self, messages: Sequence[Any], **kwargs: Any) -> SimpleNam
usage=(
SimpleNamespace(input_tokens=usage.input_tokens, output_tokens=usage.output_tokens) if usage else None
),
choices=[SimpleNamespace(finish_reason=result.finish_reason)],
)


Expand Down
54 changes: 52 additions & 2 deletions src/usersim/engine/core/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -626,7 +626,17 @@ async def acall_llm(
"""Call a model and return its reply as a message dict.

Returns role, content, and where the model supplied them
reasoning_content and tool_calls.
reasoning_content and tool_calls. Two keys describe the call rather than
the message, and are always present:

- ``finish_reason``: why the model stopped, in OpenAI's vocabulary
(``"stop"``, ``"length"``, ``"tool_calls"``, ...), or None when the
provider gave none. ``"length"`` means the reply was cut off.
- ``usage``: ``{"input_tokens": int, "output_tokens": int}``, or None when
the provider reported no token counts. One count missing reads 0.

Neither reaches a transcript: ``assistant_message`` copies only the
message's own fields.
"""
_normalize_response_format(kwargs)
facade, chat_messages, t0 = _begin_call(models, alias, messages, kwargs)
Expand All @@ -645,6 +655,40 @@ async def acall_llm(
return _finish_call(response, alias, messages, time.monotonic() - t0)


#: Anthropic's stop reasons with an exact OpenAI equivalent. Data Designer's
#: Anthropic adapter passes ``stop_reason`` through as ``finish_reason``, so
#: without this a cut-off reply reads ``max_tokens`` there and ``length``
#: everywhere else. Values without an exact equivalent pass through unchanged.
_ANTHROPIC_FINISH_REASONS = {
"end_turn": "stop",
"stop_sequence": "stop",
"max_tokens": "length",
"tool_use": "tool_calls",
}


def _reports_token_counts(usage: Any) -> bool:
"""Whether the provider reported any token count.

Data Designer returns a ``Usage`` even when a provider reports only a
total, with both counts None; that is "not reported", not zero tokens.
"""
names = ("input_tokens", "output_tokens", "prompt_tokens", "completion_tokens")
return usage is not None and any(getattr(usage, name, None) is not None for name in names)


def _finish_reason(response: Any) -> str | None:
"""The first choice's finish reason in OpenAI's vocabulary, or None.

A response without a list of choices, such as a test double, has none.
"""
choices = getattr(response, "choices", None)
reason = getattr(choices[0], "finish_reason", None) if isinstance(choices, (list, tuple)) and choices else None
if not isinstance(reason, str) or not reason:
return None
return _ANTHROPIC_FINISH_REASONS.get(reason, reason)


def _finish_call(
response: Any,
alias: str,
Expand Down Expand Up @@ -675,9 +719,15 @@ def _finish_call(
input_tokens = getattr(usage, "input_tokens", None) or getattr(usage, "prompt_tokens", None) or 0
output_tokens = getattr(usage, "output_tokens", None) or getattr(usage, "completion_tokens", None) or 0
token_tag = f", {input_tokens}+{output_tokens} tok" if (input_tokens or output_tokens) else ""
finish_reason = _finish_reason(response)
result["finish_reason"] = finish_reason
result["usage"] = (
{"input_tokens": input_tokens, "output_tokens": output_tokens} if _reports_token_counts(usage) else None
)

logger.debug(
f" |-- LLM done: {alias} in {elapsed:.1f}s ({content_len} chars / {word_cnt} words{tc_tag}{token_tag})"
f" |-- LLM done: {alias} in {elapsed:.1f}s ({content_len} chars / {word_cnt} words{tc_tag}{token_tag}"
f", finish_reason={finish_reason})"
)
_record_stat(alias, elapsed, content_len, word_cnt, input_tokens, output_tokens)
_feed_outcome_builder(alias, input_tokens, output_tokens, elapsed)
Expand Down
37 changes: 35 additions & 2 deletions src/usersim/testing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -717,7 +717,6 @@ async def hosted_parity_report(
"""
from usersim.engine.core.episode_runtime import (
ActivationRequest,
ActivationResult,
EpisodeLifecycleComplete,
)

Expand All @@ -741,7 +740,14 @@ async def hosted_parity_report(
response["reasoning_content"] = reply.message.reasoning_content
if reply.message.tool_calls:
response["tool_calls"] = copy.deepcopy(reply.message.tool_calls)
event = await hosted_runtime.advance(ActivationResult(activation_id=event.activation_id, response=response))
event = await hosted_runtime.advance(
{
"activation_id": event.activation_id,
"response": response,
"usage": _reported_usage(reply),
"finish_reason": _reported_finish_reason(reply),
}
)
if not isinstance(event, EpisodeLifecycleComplete): # pragma: no cover - defensive
raise RuntimeError("hosted episode did not complete")
hosted_row = event.result
Expand Down Expand Up @@ -796,6 +802,33 @@ async def hosted_parity_report(
_PARITY_IGNORED_COLUMNS = frozenset({"wall_clock_s", "wall_clock_s_by_alias"})


def _reported_usage(reply: Any) -> dict[str, Any] | None:
"""The reply's token counts as a host records them, so both runs count the same."""
usage = getattr(reply, "usage", None)
if usage is None:
return None
return {
"input_tokens": _first_count(usage, "input_tokens", "prompt_tokens"),
"output_tokens": _first_count(usage, "output_tokens", "completion_tokens"),
}


def _first_count(usage: Any, *names: str) -> Any:
"""The first of ``names`` the usage reports, keeping an explicit zero."""
for name in names:
value = getattr(usage, name, None)
if value is not None:
return value
return None


def _reported_finish_reason(reply: Any) -> str | None:
"""The reply's finish reason as the provider gave it; the runtime maps it as for a configured model."""
choices = getattr(reply, "choices", None)
reason = getattr(choices[0], "finish_reason", None) if isinstance(choices, (list, tuple)) and choices else None
return reason if isinstance(reason, str) else None


def _parity_value(value: Any) -> Any:
"""Normalize one column for comparison, dropping timing noise."""
if isinstance(value, str):
Expand Down
96 changes: 96 additions & 0 deletions tests/engine/core/test_episode_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -390,6 +390,102 @@ async def test_recorded_usage_reaches_the_outcome() -> None:
assert any("reasoning" in json.dumps(message) for message in json.loads(event.result["conversation_messages"]))


async def _report_the_call(self, *, models, data, cfg, state=None, seed_state=True):
"""Return what ``acall_llm`` reports about the call alongside its content."""
del data, cfg, state, seed_state
reply = await acall_llm(models, "assistant_model", [{"role": "user", "content": "Hello"}])
return {"finish_reason": reply["finish_reason"], "usage": reply["usage"]}


@pytest.fixture
def reports_call_runtime(register_test_probe) -> ProbeEpisodeRuntime:
register_test_probe("reports_call", run_dispatch=_report_the_call)
return ProbeEpisodeRuntime(
probe_type="reports_call",
persona={"first_name": "A", "last_name": "User", "age": 35},
locale="en_US",
language="English",
models={"assistant_model": HostRoleModel(model_name="host/assistant-model")},
config=ConversationSimulatorConfig(name="episode_runtime_test"),
data={},
)


@pytest.mark.parametrize(("recorded", "reported"), [("length", "length"), ("max_tokens", "length"), (None, None)])
async def test_a_recorded_finish_reason_reaches_the_probe_as_a_configured_models_would(
reports_call_runtime: ProbeEpisodeRuntime, recorded: str | None, reported: str | None
) -> None:
runtime = reports_call_runtime
request = await runtime.advance()
assert isinstance(request, ActivationRequest)

complete = await runtime.advance(
{
"activation_id": request.activation_id,
"response": {"role": "assistant", "content": "Hi"},
"usage": {"input_tokens": 5, "output_tokens": 2},
"finish_reason": recorded,
}
)

assert isinstance(complete, EpisodeLifecycleComplete)
assert complete.result["finish_reason"] == reported
assert complete.result["usage"] == {"input_tokens": 5, "output_tokens": 2}


@pytest.mark.parametrize("recorded", [None, {}, {"input_tokens": None, "output_tokens": None}])
async def test_without_recorded_token_counts_the_probe_sees_none(
reports_call_runtime: ProbeEpisodeRuntime, recorded: dict | None
) -> None:
runtime = reports_call_runtime
request = await runtime.advance()

complete = await runtime.advance(
{"activation_id": request.activation_id, "response": {"role": "assistant", "content": "Hi"}, "usage": recorded}
)

assert complete.result["usage"] is None
assert complete.result["finish_reason"] is None


def test_finish_reason_is_part_of_the_recorded_payload() -> None:
"""It round-trips through the wire form, and a non-string is rejected."""
result = ActivationResult.from_value(
{"activation_id": "a-1", "response": {"role": "assistant", "content": "x"}, "finish_reason": "length"}
)
assert result.finish_reason == "length"
assert ActivationResult.from_value(result.to_dict()) == result
with pytest.raises(TypeError, match="finish_reason"):
ActivationResult(activation_id="a-1", response={"role": "assistant", "content": "x"}, finish_reason=1)


async def test_finish_reason_and_usage_stay_out_of_the_transcript_and_later_requests() -> None:
"""The host's finish_reason and usage describe a call: no later request and
no stored message carries them."""
runtime = _open_ended_runtime()
seen: list[ActivationRequest] = []

event = await runtime.advance()
while isinstance(event, ActivationRequest):
seen.append(event)
event = await runtime.advance(
ActivationResult(
activation_id=event.activation_id,
response=_text(event),
usage=ActivationUsage(input_tokens=3, output_tokens=2),
finish_reason="length",
)
)

assert isinstance(event, EpisodeLifecycleComplete)
assert len(seen) > 2, "the episode must make later requests for this to check anything"
sent = [message for request in seen for message in request.messages]
stored = json.loads(event.result["conversation_messages"])
for message in sent + stored:
assert "finish_reason" not in message
assert "usage" not in message


async def test_concurrent_episodes_stay_isolated() -> None:
"""Episodes share a process; they must not share state."""
runtimes = [_open_ended_runtime() for _ in range(4)]
Expand Down
Loading
Loading