Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions src/usersim/engine/probes/tool_calling/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ the other probes.
run. The toolset is the row's theme.
- The simulated user writes requests the tools can serve, in the row's
language, and holds any credential a tool declares, API key or password. The
check on the opening request fails one that no offered tool can do, so the
check on the opening request fails one that no offered tool can do, one that
can't be acted on yet because it waits on something that hasn't happened,
and one that needs a file or photo the user can't give as a link, so the
user writes it again. The user asks a service about the records it holds,
such as bookings or balances, rather than describing them, and gives what
the assistant asks for that a real user would have, such as a link or a
Expand All @@ -26,12 +28,15 @@ the other probes.
- The assistant is offered the tools. A call that breaks the tool's schema gets
a 400, a call without the user's credential gets a 401, and any other call
gets a response the `api_response_model` writes for that tool. It writes from
what the service would see: the call, the calls made earlier in the
conversation with their responses, and its clock, never the conversation's
messages, so a detail the user gives reaches it only through the arguments the
assistant passes. Before the assistant sees the response, `judge_model`
checks it against the call (the tool's description, the arguments, the
earlier calls' responses, the clock, any computed value) and has a rejected
what the service would see: the call, the service's other endpoints (the
other offered tools, with their parameter types), the calls made earlier in
the conversation with their responses, and its clock, never the
conversation's messages, so a detail the user gives reaches it only through
the arguments the assistant passes. An ID or other value it returns has the
type and form the other endpoints take for it. Before the assistant sees the
response, `judge_model` checks it against the call (the tool's description,
the arguments, the other endpoints, the earlier calls' responses, the clock,
any computed value) and has a rejected
one rewritten with its reason; a response still
rejected when the attempts run out stands, and the row carries an
`unverified_tool_response` warning. A call that sets a new password leaves
Expand Down Expand Up @@ -59,7 +64,8 @@ the dates in requests and in tool results agree. The `api_response_model`'s
clock reads noon UTC on it at the user's first message and moves two minutes
with each one after, so a live score or a job's progress can move between
turns; an event after the clock has no result yet, while what is planned for
later, such as a schedule or a forecast, already exists. Each row
later, such as a schedule or a forecast, already exists. Opening hours, market
sessions and other times tied to a place follow the local time there. Each row
records the date as `today` in `conversation_metadata`. The first user message the assistant sees
opens with a line its app would attach, the date and the persona's location,
such as `[2026-11-06 · Austin, Texas, USA]`; the simulated user is told the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,15 +117,21 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
- There is **no real execution**: the `api_response_model` alias (see
[`cli/models_default.toml`](../../../../../cli/models_default.toml)) synthesizes the
tool's response from the schema, so a run needs no live service. It sees
what the service would: the call, the calls made earlier in the conversation
with their responses and the time each was made, which it keeps consistent
with, and its clock, noon UTC on the conversation's date at the user's first
message and two minutes later with each one after, up to 12:59, but none of
the conversation's messages. An event after the clock has no result yet,
while what is planned for later, such as a schedule, a listing or a
forecast, already exists; an identical call returns the same data, except
what the service would: the call, the service's other endpoints (the other
offered tools' names and parameter types), whose parameters an ID or other
value it returns fits, such as an integer ID where they take an integer, the
calls made earlier in the conversation with their responses and the time
each was made, which it keeps consistent with, and its clock, noon UTC on
the conversation's date at the user's first message and two minutes later
with each one after, up to 12:59, but none of
the conversation's messages. Opening hours, market sessions and other times
tied to a place follow the local time there. An event after the clock has no
result yet, while what is planned for later, such as a schedule, a listing
or a forecast, already exists; an identical call returns the same data, except
that what moves with time, such as a live score, a job's progress, stock or
a price, may have moved since it was made. A tool with a parameter naming one
a price, may have moved since it was made, and a tool that returns random or
newly generated items by design, such as a random image or a generated
password, returns new ones each time. A tool with a parameter naming one
of the user's own things (a tracking, order, booking or reservation number,
an invoice or a receipt) also gets the service's records on the person,
their name and where they live, which that thing carries, while an argument
Expand All @@ -147,9 +153,12 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
assistant sees it: that it answers the call, agrees with the earlier calls'
responses and the clock, gives no result for an event later than the clock
while letting a schedule or forecast for later stand,
computes values correctly, and holds no placeholder
such as an `example.com` link. It checks real-world facts only where the
arguments or the tool's description fix them. A rejected response is
computes values correctly (a field that only describes the response, such
as a count of the items it lists, is not grounds for rejecting it), returns
IDs and values of the type and form the service's other endpoints take, and
holds no placeholder such as an `example.com` link. It checks real-world
facts only where the arguments or the tool's description fix them. A
rejected response is
rewritten with the judge's reason, within `max_response_attempts` (the
probe's `judge_responses` setting turns this off). The judge is the models file's `api_response_judge_model` (the
probe's `response_judge_model_alias` setting), so these checks can run on
Expand All @@ -162,7 +171,9 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
`n_assistant_ungrounded`. The judge also sees the tools the assistant was
offered, and fails a reply that says it can't do what one of them does,
rather than asking the user for what the call is missing, or calls its
tools' results unreliable without a reason. It judges only what
tools' results unreliable without a reason; declining for a legal, safety
or privacy reason, or saying accurately that no offered tool does what is
asked, is not that failure. It judges only what
the reply in front of it says: an earlier reply's error counts against that
reply alone, unless this one repeats it. With `--max-assistant-attempts`
above 1, the rejected turn is undone, tool calls included, and regenerated.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,21 +86,31 @@ tool_simulator_prompt: |-
It is {now}. An event later than that has not happened, so it has no result, score or reading yet;
what is planned for later, such as a schedule, showtime, listing, forecast, booking or price, already
exists.
Opening hours, market sessions and other times tied to a place follow the local time there, converted
from that UTC time.

Tool specification:
{tool_spec}

Tool call arguments:
{tool_args}

The service's other endpoints:
<OTHER_ENDPOINTS>
{other_endpoints}
</OTHER_ENDPOINTS>
An ID or other value the response returns has the type and form these endpoints take for it, such as an
integer ID where they take an integer.

The calls this caller made earlier, each with the time it was made and what the service returned, are
the service's own records:
<EARLIER_CALLS>
{earlier_calls}
</EARLIER_CALLS>
An identical call returns the same data unless a call in between changed it, and anything an
earlier response returned keeps its values; only what moves with time, such as a live score, a job's
progress, stock or a price, may have moved since then.
progress, stock or a price, may have moved since then. A tool that returns random or newly generated
items by design, such as a random image or a generated password, returns new ones each time.

Write it as the real service would:
- Links use the service's own domain or another realistic one, never example.com, another
Expand All @@ -127,13 +137,20 @@ tool_simulator_judge_prompt: |-
You are checking a simulated API response before an AI assistant sees it.

It is {now}.
Opening hours, market sessions and other times tied to a place follow the local time there, converted
from that UTC time.

Tool specification:
{tool_spec}

Tool call arguments:
{tool_args}
{credentials}{caller}
The service's other endpoints:
<OTHER_ENDPOINTS>
{other_endpoints}
</OTHER_ENDPOINTS>

The calls this caller made earlier, each with the time it was made and what the service returned:
<EARLIER_CALLS>
{earlier_calls}
Expand All @@ -149,9 +166,13 @@ tool_simulator_judge_prompt: |-
- The response is what this tool returns for these arguments: it concerns the entity, place, dates, units and ranges the arguments ask for, and its fields fit the tool's description.
- It is consistent with itself, with the earlier calls' responses and with the current time: an identical
call returns the same data unless a call in between changed it, or time has passed and the data is the
kind that moves with it, such as a live score, a job's progress, stock or a price. Anything planned for
kind that moves with it, such as a live score, a job's progress, stock or a price. A tool that returns
random or newly generated items by design returns new ones each time. Anything planned for
after the current time, such as a schedule, a listing, a forecast or a booking, is fine.
- Every value the tool computes (a total, a conversion, a date difference, a formula) is correct for the arguments.
- A field that only describes the response itself, such as a count of the items it lists or an offset
derived from its other fields, is not grounds for failure.
- An ID or other value it returns that the service's other endpoints take has the type and form they take for it.
- It reads like real API output: no placeholder values (an example.com or other example.* host is one)
and no commentary around the JSON.
- Real-world facts are checked only where the arguments or the tool's description fix them, such as a
Expand All @@ -162,6 +183,7 @@ tool_simulator_judge_prompt: |-
- It answers a different request than the arguments make, or contradicts the tool's description.
- It contradicts itself, an earlier response or the current time, or gives a result, score or reading for an event after the current time.
- A computed value is wrong.
- It returns an ID or other value in a type or form the service's other endpoints don't take for it.
- It holds placeholder values or commentary.
- It is an error the call does not warrant.
</RUBRIC>
Expand Down Expand Up @@ -242,6 +264,8 @@ tool_relevance_failure: |
- The request can be fully answered from general knowledge or opinion alone,
without needing any of the available tools.
- The request asks for something none of the available tools can do.
- The request can't be acted on yet, because it waits on something that hasn't
happened, or it needs a file or photo the user can't give as a link.
assistant_judge_prompt: |-
You are an expert evaluator. Judge the assistant's response in this conversation.

Expand Down Expand Up @@ -274,7 +298,7 @@ assistant_judge_prompt: |-
- The response does not address the user's question or request.
- The response is unintelligible, empty, or a non-sequitur that ignores the conversation context.
- The response states a figure, name, date or other result that no tool response in the conversation contains, contradicts a tool response, or says it performed an action, such as a booking, a change or a lookup, that no tool call in the conversation performed.
- The response says it cannot look up or do something one of the assistant's tools does, rather than asking the user for what the call is missing, or calls results its tool calls returned unreliable without a reason the conversation shows.
- The response says it cannot look up or do something one of the assistant's tools does, rather than asking the user for what the call is missing, or calls results its tool calls returned unreliable without a reason the conversation shows. Declining for a legal, safety or privacy reason, or saying accurately that no offered tool does what is asked, is not this failure.

</RUBRIC>

Expand Down
32 changes: 30 additions & 2 deletions src/usersim/engine/probes/tool_calling/probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,26 @@ def _build_tool_context(tool_subset: list) -> str:
return "\n".join(lines) if lines else "No tools available."


def _other_endpoints(tool_subset: list, tool_name: str) -> str:
"""The offered tools other than ``tool_name``, one per line as ``- name(parameter: type, ...)``, or ``None.``.

The tool simulator and its judge read them as the service's other endpoints, so an ID or other value a
response returns has the type and form the endpoints that take it accept.
"""
lines = []
for tool_data in tool_subset:
spec = _function_spec(tool_data)
name = spec.get("name")
if not name or name == tool_name:
continue
properties = (spec.get("parameters") or {}).get("properties") or {}
parameters = ", ".join(
f"{parameter}: {(schema or {}).get('type', 'any')}" for parameter, schema in properties.items()
)
lines.append(f"- {name}({parameters})")
return "\n".join(lines) if lines else "None."


def _row_identity(data: dict) -> str:
"""The row's identity, which seeds its tool draw and the credentials its user holds.

Expand Down Expand Up @@ -366,12 +386,14 @@ async def _judge_tool_response(
credential_parameters: tuple[str, ...],
caller: str = "",
judge_alias: str = MODEL_JUDGE,
other_endpoints: str = "None.",
) -> tuple[bool, str] | None:
"""The ``judge_alias`` model's verdict on one simulated response, ``(passed, explanation)``; None when it can't be read."""
prompt = tool_simulator_judge_prompt().format(
now=now,
tool_spec=json.dumps(tool_spec, indent=2, default=str),
tool_args=tool_args,
other_endpoints=other_endpoints,
credentials=(
"\n"
+ tool_simulator_judge_credentials().format(
Expand Down Expand Up @@ -401,10 +423,13 @@ async def _simulate_tool_response(
judge: bool = False,
max_attempts: int = 2,
judge_alias: str = MODEL_JUDGE,
other_endpoints: str = "None.",
) -> SimulatedResponse:
"""Use the strong frontier model to simulate a realistic API response.

The simulator sees what the service would: the call, the calls made
The simulator sees what the service would: the call, the service's other
endpoints (``other_endpoints``, from ``_other_endpoints``), whose
parameters the IDs and values it returns fit, the calls made
earlier in the conversation with their responses (``_earlier_calls``),
which it keeps consistent with, ``now``, the service's clock, which moves
as the conversation goes on, after which no event has a result yet though
Expand Down Expand Up @@ -433,6 +458,7 @@ async def _simulate_tool_response(
prompt = tool_simulator_prompt().format(
tool_spec=json.dumps(tool_spec, indent=2, default=str),
tool_args=tool_args,
other_endpoints=other_endpoints,
earlier_calls=earlier_calls,
now=now,
)
Expand Down Expand Up @@ -471,6 +497,7 @@ async def _simulate_tool_response(
credential_parameters=credential_parameters,
caller=caller,
judge_alias=judge_alias,
other_endpoints=other_endpoints,
)
except Exception as e: # a judge outage leaves the response standing, flagged
logger.warning(" |-- %s raised on a simulated response: %s: %s", judge_alias, type(e).__name__, e)
Expand Down Expand Up @@ -527,7 +554,7 @@ class ToolCallingProbe(ToolExecutionMixin, ToolCallingMixin, BaseProbe):
label = "tool_calling"
# Bump on any semantic change to the probe's prompts or task derivation.
# One variant: the tool category is the row's ``toolset_category``.
prompt_version = "v1.11"
prompt_version = "v1.12"
assets_root = Path(__file__).parent / "assets"
# The toolset catalog: each row draws one toolset and reads its tools from
# the ``tools`` column. The toolset is also the row's theme, so the probe
Expand Down Expand Up @@ -813,6 +840,7 @@ async def execute_tool_call(
judge=self.options.judge_responses,
max_attempts=self.options.max_response_attempts,
judge_alias=judge_alias,
other_endpoints=_other_endpoints(self.tool_subset, name),
)
simulated_response = simulated.content
self._record_simulation(
Expand Down
Loading
Loading