Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions src/usersim/engine/probes/tool_calling/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,11 @@ used by a reply that is not JSON or that the judge rejects (default `2`);
`share_user_context`, whether the assistant sees the date and where the user
is (default `true`).
The simulated user and the `api_response_model` both work from that date, so
the dates in requests and in tool results agree; the `api_response_model`'s
clock reads noon UTC on it, and nothing later has happened yet. Each row
the dates in requests and in tool results agree. The `api_response_model`'s
clock reads noon UTC on it at the user's first message and moves two minutes
with each one after, so a live score or a job's progress can move between
turns; an event after the clock has no result yet, while what is planned for
later, such as a schedule or a forecast, already exists. Each row
records the date as `today` in `conversation_metadata`. The first user message the assistant sees
opens with a line its app would attach, the date and the persona's location,
such as `[2026-11-06 · Austin, Texas, USA]`; the simulated user is told the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,14 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
[`cli/models_default.toml`](../../../../../cli/models_default.toml)) synthesizes the
tool's response from the schema, so a run needs no live service. It sees
what the service would: the call, the calls made earlier in the conversation
with their responses, which it keeps consistent with, and its clock, noon UTC
on the conversation's date, before which everything it returns has happened,
but none of the conversation's messages. A tool with a parameter naming one
with their responses and the time each was made, which it keeps consistent
with, and its clock, noon UTC on the conversation's date at the user's first
message and two minutes later with each one after, up to 12:59, but none of
the conversation's messages. An event after the clock has no result yet,
while what is planned for later, such as a schedule, a listing or a
forecast, already exists; an identical call returns the same data, except
that what moves with time, such as a live score, a job's progress, stock or
a price, may have moved since it was made. A tool with a parameter naming one
of the user's own things (a tracking, order, booking or reservation number,
an invoice or a receipt) also gets the service's records on the person,
their name and where they live, which that thing carries, while an argument
Expand All @@ -140,7 +145,8 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
pages, the field that leads to the rest.
- A judge checks each synthesized response against the call before the
assistant sees it: that it answers the call, agrees with the earlier calls'
responses and the clock, gives no result for anything later than the clock,
responses and the clock, gives no result for an event later than the clock
while letting a schedule or forecast for later stand,
computes values correctly, and holds no placeholder
such as an `example.com` link. It checks real-world facts only where the
arguments or the tool's description fix them. A rejected response is
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,21 +83,24 @@ tool_simulator_prompt: |-
that the tool would return. The response should be consistent with the tool's description
and the arguments provided.

It is {now}. Nothing after that has happened yet: a match, release or reading later than that has
no result.
It is {now}. An event later than that has not happened, so it has no result, score or reading yet;
what is planned for later, such as a schedule, showtime, listing, forecast, booking or price, already
exists.

Tool specification:
{tool_spec}

Tool call arguments:
{tool_args}

The calls this caller made earlier, with what the service returned, are the service's own records:
The calls this caller made earlier, each with the time it was made and what the service returned, are
the service's own records:
<EARLIER_CALLS>
{earlier_calls}
</EARLIER_CALLS>
An identical call returns the same data unless a call in between changed it, and anything an
earlier response returned keeps its values.
earlier response returned keeps its values; only what moves with time, such as a live score, a job's
progress, stock or a price, may have moved since then.

Write it as the real service would:
- Links use the service's own domain or another realistic one, never example.com, another
Expand Down Expand Up @@ -131,7 +134,7 @@ tool_simulator_judge_prompt: |-
Tool call arguments:
{tool_args}
{credentials}{caller}
The calls this caller made earlier, with what the service returned:
The calls this caller made earlier, each with the time it was made and what the service returned:
<EARLIER_CALLS>
{earlier_calls}
</EARLIER_CALLS>
Expand All @@ -145,7 +148,9 @@ tool_simulator_judge_prompt: |-
Criteria for success:
- The response is what this tool returns for these arguments: it concerns the entity, place, dates, units and ranges the arguments ask for, and its fields fit the tool's description.
- It is consistent with itself, with the earlier calls' responses and with the current time: an identical
call returns the same data unless a call in between changed it.
call returns the same data unless a call in between changed it, or time has passed and the data is the
kind that moves with it, such as a live score, a job's progress, stock or a price. Anything planned for
after the current time, such as a schedule, a listing, a forecast or a booking, is fine.
- Every value the tool computes (a total, a conversion, a date difference, a formula) is correct for the arguments.
- It reads like real API output: no placeholder values (an example.com or other example.* host is one)
and no commentary around the JSON.
Expand All @@ -155,7 +160,7 @@ tool_simulator_judge_prompt: |-

Criteria for failure:
- It answers a different request than the arguments make, or contradicts the tool's description.
- It contradicts itself, an earlier response or the current time, or gives a result for something after the current time.
- It contradicts itself, an earlier response or the current time, or gives a result, score or reading for an event after the current time.
- A computed value is wrong.
- It holds placeholder values or commentary.
- It is an error the call does not warrant.
Expand Down
48 changes: 35 additions & 13 deletions src/usersim/engine/probes/tool_calling/probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,24 +257,29 @@ def _argument_values(arguments: str) -> list[str]:


def _earlier_calls(conversation_messages: list, tool_name: str, arguments: str) -> str:
"""The calls made earlier in the conversation, each with its arguments and response: the service's records.
"""The calls made earlier in the conversation, each with its arguments, time and response: the service's records.

Every call is listed, in order. Responses go in whole up to
Every call is listed, in order, with the clock it was made at
(``_clock``). Responses go in whole up to
``_EARLIER_CALLS_CHARS``: first the response to the same call
(``tool_name`` with ``arguments``), then those naming one of its argument
values, then the rest of ``tool_name``'s, then everything else, each
newest first. A response past that shows as its call alone.
"""
calls: dict[Any, tuple[str, str]] = {}
records: list[tuple[str, str, str, str]] = []
user_messages = 0
for message in conversation_messages:
if message.get("role") == "assistant":
if message.get("role") == "user":
user_messages += 1
elif message.get("role") == "assistant":
for call in message.get("tool_calls") or []:
function = call.get("function", {})
calls[call.get("id")] = (function.get("name", "?"), _arguments_text(function.get("arguments", "{}")))
elif message.get("role") == "tool":
name, args = calls.get(message.get("tool_call_id"), ("?", "{}"))
records.append((f"{name}({args})", name, _canonical_arguments(args), message.get("content") or ""))
call = f"{name}({args}) at {_clock(user_messages)} UTC"
records.append((call, name, _canonical_arguments(args), message.get("content") or ""))
if not records:
return "None."
same_call = (tool_name, _canonical_arguments(arguments))
Expand Down Expand Up @@ -305,13 +310,29 @@ def _format_day(day: date) -> str:
return f"{_WEEKDAYS[day.weekday()]}, {day.isoformat()}"


def _format_now(day: date) -> str:
"""The simulated service's clock on ``day``, such as ``Friday, 2026-11-06, 12:00 UTC``.
#: How far the simulated service's clock moves with each user message after the first: the time a person takes
#: to read a reply and write back.
_MINUTES_PER_USER_MESSAGE = 2

Noon UTC falls on ``day`` in every time zone from UTC-11 to UTC+11, so the
clock never disagrees with the date the user and the assistant work from.

def _user_messages(conversation_messages: list) -> int:
return sum(1 for message in conversation_messages if message.get("role") == "user")


def _clock(user_messages: int) -> str:
"""The simulated service's clock, as ``12:04``, once the user has sent ``user_messages`` messages.

Noon UTC at the first, moving ``_MINUTES_PER_USER_MESSAGE`` with each one
after it and stopping at 12:59, so the clock stays on the conversation's
date in every time zone from UTC-11 to UTC+11, the date the user and the
assistant work from.
"""
return f"{_format_day(day)}, 12:00 UTC"
return f"12:{min(_MINUTES_PER_USER_MESSAGE * max(user_messages - 1, 0), 59):02d}"


def _format_now(day: date, user_messages: int = 1) -> str:
"""The simulated service's date and clock, such as ``Friday, 2026-11-06, 12:02 UTC``."""
return f"{_format_day(day)}, {_clock(user_messages)} UTC"


#: What a call answers when no attempt produced JSON.
Expand Down Expand Up @@ -385,8 +406,9 @@ async def _simulate_tool_response(

The simulator sees what the service would: the call, the calls made
earlier in the conversation with their responses (``_earlier_calls``),
which it keeps consistent with, ``now``, the service's clock on the
conversation's date, before which everything it returns has happened,
which it keeps consistent with, ``now``, the service's clock, which moves
as the conversation goes on, after which no event has a result yet though
what is planned for later, such as a schedule, already exists,
and ``caller``, the service's records on the person, which the user's
own things carry and nothing else does. The conversation's
messages reach neither it nor its judge. The probe checks a call's
Expand Down Expand Up @@ -505,7 +527,7 @@ class ToolCallingProbe(ToolExecutionMixin, ToolCallingMixin, BaseProbe):
label = "tool_calling"
# Bump on any semantic change to the probe's prompts or task derivation.
# One variant: the tool category is the row's ``toolset_category``.
prompt_version = "v1.10"
prompt_version = "v1.11"
assets_root = Path(__file__).parent / "assets"
# The toolset catalog: each row draws one toolset and reads its tools from
# the ``tools`` column. The toolset is also the row's theme, so the probe
Expand Down Expand Up @@ -785,7 +807,7 @@ async def execute_tool_call(
tool_spec,
tc,
state.messages,
now=_format_now(self.options.today),
now=_format_now(self.options.today, _user_messages(state.messages)),
credential_parameters=tuple(declared),
caller=self._caller_records(tool_spec),
judge=self.options.judge_responses,
Expand Down
19 changes: 12 additions & 7 deletions src/usersim/engine/probes/tool_calling/prompts.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,21 +107,24 @@
that the tool would return. The response should be consistent with the tool's description
and the arguments provided.

It is {now}. Nothing after that has happened yet: a match, release or reading later than that has
no result.
It is {now}. An event later than that has not happened, so it has no result, score or reading yet;
what is planned for later, such as a schedule, showtime, listing, forecast, booking or price, already
exists.

Tool specification:
{tool_spec}

Tool call arguments:
{tool_args}

The calls this caller made earlier, with what the service returned, are the service's own records:
The calls this caller made earlier, each with the time it was made and what the service returned, are
the service's own records:
<EARLIER_CALLS>
{earlier_calls}
</EARLIER_CALLS>
An identical call returns the same data unless a call in between changed it, and anything an
earlier response returned keeps its values.
earlier response returned keeps its values; only what moves with time, such as a live score, a job's
progress, stock or a price, may have moved since then.

Write it as the real service would:
- Links use the service's own domain or another realistic one, never example.com, another
Expand Down Expand Up @@ -174,7 +177,7 @@
Tool call arguments:
{tool_args}
{credentials}{caller}
The calls this caller made earlier, with what the service returned:
The calls this caller made earlier, each with the time it was made and what the service returned:
<EARLIER_CALLS>
{earlier_calls}
</EARLIER_CALLS>
Expand All @@ -188,7 +191,9 @@
Criteria for success:
- The response is what this tool returns for these arguments: it concerns the entity, place, dates, units and ranges the arguments ask for, and its fields fit the tool's description.
- It is consistent with itself, with the earlier calls' responses and with the current time: an identical
call returns the same data unless a call in between changed it.
call returns the same data unless a call in between changed it, or time has passed and the data is the
kind that moves with it, such as a live score, a job's progress, stock or a price. Anything planned for
after the current time, such as a schedule, a listing, a forecast or a booking, is fine.
- Every value the tool computes (a total, a conversion, a date difference, a formula) is correct for the arguments.
- It reads like real API output: no placeholder values (an example.com or other example.* host is one)
and no commentary around the JSON.
Expand All @@ -198,7 +203,7 @@

Criteria for failure:
- It answers a different request than the arguments make, or contradicts the tool's description.
- It contradicts itself, an earlier response or the current time, or gives a result for something after the current time.
- It contradicts itself, an earlier response or the current time, or gives a result, score or reading for an event after the current time.
- A computed value is wrong.
- It holds placeholder values or commentary.
- It is an error the call does not warrant.
Expand Down
2 changes: 1 addition & 1 deletion tests/probes/tool_calling/contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@

CONTRACTS = {
"tool_calling": ProbeContract(
trajectory_id="d537ad1a3fd6aa2a",
trajectory_id="07f08aa06b2909b8",
# A conversation with no tool calls completed its protocol, and the evaluator scores it.
succeeds_on=None,
parity=(ParityCase(1003, {"tools": [_WEATHER_TOOL]}),),
Expand Down
33 changes: 32 additions & 1 deletion tests/probes/tool_calling/test_probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import types
from collections.abc import Callable
from contextlib import ExitStack
from datetime import date
from unittest.mock import patch

import pytest
Expand All @@ -25,6 +26,7 @@
ToolCallingProbe,
_build_tool_context,
_find_tool_spec,
_format_now,
_format_tools_for_api,
_simulate_tool_response,
)
Expand Down Expand Up @@ -620,7 +622,36 @@ def test_the_user_is_told_what_the_mark_means(self, sample_tools):

class TestToday:
"""The simulated user, the assistant and the API response simulator share one date, which the simulator's clock
reads at noon UTC; the assistant also sees the user's place."""
reads at noon UTC at the user's first message and later with each one after; the assistant also sees the user's
place."""

@pytest.mark.parametrize(
("user_messages", "clock"), [(0, "12:00"), (1, "12:00"), (2, "12:02"), (4, "12:06"), (100, "12:59")]
)
def test_the_clock_moves_two_minutes_with_each_user_message_and_stays_on_the_date(self, user_messages, clock):
assert _format_now(date(2026, 11, 6), user_messages) == f"Friday, 2026-11-06, {clock} UTC"

async def test_a_call_after_the_users_second_message_reads_the_moved_clock(self, sample_tools):
probe, _data, _cfg = _concrete_tool_probe(sample_tools, judge_responses=False)
state = ConversationState()
state.messages += [
{"role": "user", "content": "Weather?"},
{"role": "assistant", "content": "Where are you?"},
{"role": "user", "content": "Tokyo."},
]
prompts = []

async def fake_call_llm(models, alias, messages, **kwargs):
prompts.append(messages[0]["content"])
return {"content": '{"temp": 22}'}

tool_call = {"id": "c", "function": {"name": "get_weather", "arguments": '{"location": "Tokyo"}'}}
with patch("usersim.engine.probes.tool_calling.probe.acall_llm", side_effect=fake_call_llm):
await probe.execute_tool_call(
"get_weather", {"location": "Tokyo"}, tool_call, state, {}, turn_idx=1, call_idx=0
)

assert "It is Friday, 2026-11-06, 12:02 UTC." in prompts[0]

async def test_the_user_and_the_tool_simulator_share_the_runs_date(self, sample_tools):
result, _assistant, user_inputs, _judge, api_inputs = await _run_concrete_tool_probe(sample_tools)
Expand Down
Loading
Loading