Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions src/usersim/engine/probes/tool_calling/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ The simulated user and the `api_response_model` both work from that date, so
the dates in requests and in tool results agree. The `api_response_model`'s
clock reads noon UTC on it at the user's first message and moves two minutes
with each one after, so a live score or a job's progress can move between
turns; an event after the clock has no result yet, while what is planned for
later, such as a schedule or a forecast, already exists. Opening hours, market
turns, and gives the Unix time of that moment, which timestamps count from;
an event after the clock has no result yet, while what is planned for later,
such as a schedule or a forecast, already exists. Opening hours, market
sessions and other times tied to a place follow the local time there. Each row
records the date as `today` in `conversation_metadata`. The first user message the assistant sees
opens with a line its app would attach, the date and the persona's location,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -123,15 +123,17 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
calls made earlier in the conversation with their responses and the time
each was made, which it keeps consistent with, and its clock, noon UTC on
the conversation's date at the user's first message and two minutes later
with each one after, up to 12:59, but none of
with each one after, up to 12:59, with the Unix time of that moment for
timestamps to count from, but none of
the conversation's messages. Opening hours, market sessions and other times
tied to a place follow the local time there. An event after the clock has no
result yet, while what is planned for later, such as a schedule, a listing
or a forecast, already exists; an identical call returns the same data, except
that what moves with time, such as a live score, a job's progress, stock or
a price, may have moved since it was made, and a tool that returns random or
newly generated items by design, such as a random image or a generated
password, returns new ones each time. A tool with a parameter naming one
password, returns new ones each time. An ID a response returned names the
same record in every later response. A tool with a parameter naming one
of the user's own things (a tracking, order, booking or reservation number,
an invoice or a receipt) also gets the service's records on the person,
their name and where they live, which that thing carries, while an argument
Expand All @@ -151,8 +153,10 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
pages, the field that leads to the rest.
- A judge checks each synthesized response against the call before the
assistant sees it: that it answers the call, agrees with the earlier calls'
responses and the clock, gives no result for an event later than the clock
while letting a schedule or forecast for later stand,
responses and the clock (a timestamp counted from its Unix time, and an ID
an earlier response returned still naming the same record, while a tool
that returns random items may repeat one), gives no result for an event
later than the clock while letting a schedule or forecast for later stand,
computes values correctly (a field that only describes the response, such
as a count of the items it lists, is not grounds for rejecting it), returns
IDs and values of the type and form the service's other endpoints take, and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ tool_simulator_prompt: |-
what is planned for later, such as a schedule, showtime, listing, forecast, booking or price, already
exists.
Opening hours, market sessions and other times tied to a place follow the local time there, converted
from that UTC time.
from that UTC time. A Unix timestamp counts from the Unix time given there, 86,400 seconds a day.

Tool specification:
{tool_spec}
Expand All @@ -113,6 +113,8 @@ tool_simulator_prompt: |-
earlier response returned keeps its values; only what moves with time, such as a live score, a job's
progress, stock or a price, may have moved since then. A tool that returns random or newly generated
items by design, such as a random image or a generated password, returns new ones each time.
An ID an earlier response returned names the same record in every response: its name, title and
other fields stay as returned, and no other record takes that ID.

Write it as the real service would:
- Links use the service's own domain or another realistic one, never example.com, another
Expand Down Expand Up @@ -140,7 +142,8 @@ tool_simulator_judge_prompt: |-

It is {now}.
Opening hours, market sessions and other times tied to a place follow the local time there, converted
from that UTC time.
from that UTC time. Check a Unix timestamp by counting from the Unix time given there, 86,400 seconds a
day, not by working one out from the calendar.

Tool specification:
{tool_spec}
Expand Down Expand Up @@ -169,7 +172,7 @@ tool_simulator_judge_prompt: |-
- It is consistent with itself, with the earlier calls' responses and with the current time: an identical
call returns the same data unless a call in between changed it, or time has passed and the data is the
kind that moves with it, such as a live score, a job's progress, stock or a price. A tool that returns
random or newly generated items by design returns new ones each time. Anything planned for
random or newly generated items by design may return new ones or repeat one. Anything planned for
after the current time, such as a schedule, a listing, a forecast or a booking, is fine.
- Every value the tool computes (a total, a conversion, a date difference, a formula) is correct for the arguments.
- A field that only describes the response itself, such as a count of the items it lists or an offset
Expand All @@ -184,6 +187,7 @@ tool_simulator_judge_prompt: |-
Criteria for failure:
- It answers a different request than the arguments make, or contradicts the tool's description.
- It contradicts itself, an earlier response or the current time, or gives a result, score or reading for an event after the current time.
- It gives an ID an earlier response returned to a different record, or changes that record's name, title or other fields that don't move with time.
- A computed value is wrong.
- It returns an ID or other value in a type or form the service's other endpoints don't take for it.
- It holds placeholder values or commentary.
Expand Down
14 changes: 10 additions & 4 deletions src/usersim/engine/probes/tool_calling/probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
import random
import re
from dataclasses import dataclass, field
from datetime import date
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any

Expand Down Expand Up @@ -351,8 +351,14 @@ def _clock(user_messages: int) -> str:


def _format_now(day: date, user_messages: int = 1) -> str:
"""The simulated service's date and clock, such as ``Friday, 2026-11-06, 12:02 UTC``."""
return f"{_format_day(day)}, {_clock(user_messages)} UTC"
"""The simulated service's date and clock, such as ``Friday, 2026-11-06, 12:02 UTC (Unix time 1793966520)``.

The Unix time is given so a timestamp in a response can be counted from it, not worked out from the date.
"""
clock = _clock(user_messages)
hour, minute = (int(part) for part in clock.split(":"))
unix = int(datetime(day.year, day.month, day.day, hour, minute, tzinfo=timezone.utc).timestamp())
return f"{_format_day(day)}, {clock} UTC (Unix time {unix})"


#: What a call answers when no attempt produced JSON.
Expand Down Expand Up @@ -554,7 +560,7 @@ class ToolCallingProbe(ToolExecutionMixin, ToolCallingMixin, BaseProbe):
label = "tool_calling"
# Bump on any semantic change to the probe's prompts or task derivation.
# One variant: the tool category is the row's ``toolset_category``.
prompt_version = "v1.13"
prompt_version = "v1.14"
assets_root = Path(__file__).parent / "assets"
# The toolset catalog: each row draws one toolset and reads its tools from
# the ``tools`` column. The toolset is also the row's theme, so the probe
Expand Down
10 changes: 7 additions & 3 deletions src/usersim/engine/probes/tool_calling/prompts.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@
what is planned for later, such as a schedule, showtime, listing, forecast, booking or price, already
exists.
Opening hours, market sessions and other times tied to a place follow the local time there, converted
from that UTC time.
from that UTC time. A Unix timestamp counts from the Unix time given there, 86,400 seconds a day.

Tool specification:
{tool_spec}
Expand All @@ -137,6 +137,8 @@
earlier response returned keeps its values; only what moves with time, such as a live score, a job's
progress, stock or a price, may have moved since then. A tool that returns random or newly generated
items by design, such as a random image or a generated password, returns new ones each time.
An ID an earlier response returned names the same record in every response: its name, title and
other fields stay as returned, and no other record takes that ID.

Write it as the real service would:
- Links use the service's own domain or another realistic one, never example.com, another
Expand Down Expand Up @@ -183,7 +185,8 @@

It is {now}.
Opening hours, market sessions and other times tied to a place follow the local time there, converted
from that UTC time.
from that UTC time. Check a Unix timestamp by counting from the Unix time given there, 86,400 seconds a
day, not by working one out from the calendar.

Tool specification:
{tool_spec}
Expand Down Expand Up @@ -212,7 +215,7 @@
- It is consistent with itself, with the earlier calls' responses and with the current time: an identical
call returns the same data unless a call in between changed it, or time has passed and the data is the
kind that moves with it, such as a live score, a job's progress, stock or a price. A tool that returns
random or newly generated items by design returns new ones each time. Anything planned for
random or newly generated items by design may return new ones or repeat one. Anything planned for
after the current time, such as a schedule, a listing, a forecast or a booking, is fine.
- Every value the tool computes (a total, a conversion, a date difference, a formula) is correct for the arguments.
- A field that only describes the response itself, such as a count of the items it lists or an offset
Expand All @@ -227,6 +230,7 @@
Criteria for failure:
- It answers a different request than the arguments make, or contradicts the tool's description.
- It contradicts itself, an earlier response or the current time, or gives a result, score or reading for an event after the current time.
- It gives an ID an earlier response returned to a different record, or changes that record's name, title or other fields that don't move with time.
- A computed value is wrong.
- It returns an ID or other value in a type or form the service's other endpoints don't take for it.
- It holds placeholder values or commentary.
Expand Down
2 changes: 1 addition & 1 deletion tests/probes/tool_calling/contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@

CONTRACTS = {
"tool_calling": ProbeContract(
trajectory_id="18e99dc0c369e6c1",
trajectory_id="bf2fc50c4b1f1bb2",
# A conversation with no tool calls completed its protocol, and the evaluator scores it.
succeeds_on=None,
parity=(ParityCase(1003, {"tools": [_WEATHER_TOOL]}),),
Expand Down
58 changes: 49 additions & 9 deletions tests/probes/tool_calling/test_probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
import types
from collections.abc import Callable
from contextlib import ExitStack
from datetime import date
from datetime import date, datetime, timezone
from unittest.mock import patch

import pytest
Expand Down Expand Up @@ -681,10 +681,22 @@ class TestToday:
place."""

@pytest.mark.parametrize(
("user_messages", "clock"), [(0, "12:00"), (1, "12:00"), (2, "12:02"), (4, "12:06"), (100, "12:59")]
("user_messages", "clock", "unix"),
[
(0, "12:00", 1793966400),
(1, "12:00", 1793966400),
(2, "12:02", 1793966520),
(4, "12:06", 1793966760),
(100, "12:59", 1793969940),
],
)
def test_the_clock_moves_two_minutes_with_each_user_message_and_stays_on_the_date(self, user_messages, clock):
assert _format_now(date(2026, 11, 6), user_messages) == f"Friday, 2026-11-06, {clock} UTC"
def test_the_clock_moves_two_minutes_with_each_user_message_and_stays_on_the_date(self, user_messages, clock, unix):
assert _format_now(date(2026, 11, 6), user_messages) == f"Friday, 2026-11-06, {clock} UTC (Unix time {unix})"

def test_the_unix_time_is_the_clocks_moment(self):
"""A service that counts a timestamp from the Unix time lands on the clock's date and minute."""
unix = int(_format_now(date(2026, 11, 6), 3).rsplit(" ", 1)[1].rstrip(")"))
assert datetime.fromtimestamp(unix, timezone.utc) == datetime(2026, 11, 6, 12, 4, tzinfo=timezone.utc)

async def test_a_call_after_the_users_second_message_reads_the_moved_clock(self, sample_tools):
probe, _data, _cfg = _concrete_tool_probe(sample_tools, judge_responses=False)
Expand All @@ -706,7 +718,7 @@ async def fake_call_llm(models, alias, messages, **kwargs):
"get_weather", {"location": "Tokyo"}, tool_call, state, {}, turn_idx=1, call_idx=0
)

assert "It is Friday, 2026-11-06, 12:02 UTC." in prompts[0]
assert "It is Friday, 2026-11-06, 12:02 UTC (Unix time 1793966520)." in prompts[0]

async def test_the_simulator_and_its_judge_read_times_tied_to_a_place_in_its_local_time(self):
_result, api_prompts, judge_prompts = await _simulate_with_judge()
Expand All @@ -719,7 +731,7 @@ async def test_the_user_and_the_tool_simulator_share_the_runs_date(self, sample_
result, _assistant, user_inputs, _judge, api_inputs = await _run_concrete_tool_probe(sample_tools)

assert "Today is Friday, 2026-11-06." in user_inputs[0][0]["content"]
assert "It is Friday, 2026-11-06, 12:00 UTC." in api_inputs[0][0]["content"]
assert "It is Friday, 2026-11-06, 12:00 UTC (Unix time 1793966400)." in api_inputs[0][0]["content"]
assert json.loads(result["conversation_metadata"])["today"] == "2026-11-06"

async def test_the_assistant_sees_the_date_and_the_users_place_on_the_first_message(self, sample_tools):
Expand Down Expand Up @@ -845,11 +857,39 @@ def _tool(tool_name: str, **parameter_types: str) -> dict:
return {"tool": {"type": "function", "function": {"name": tool_name, "parameters": {"properties": properties}}}}


async def test_a_tool_that_returns_random_or_generated_items_returns_new_ones_each_time():
async def test_a_tool_that_returns_random_or_generated_items_returns_new_ones_and_may_repeat_one():
"""The simulator varies what a random tool returns; the judge doesn't fail a repeat, which a real one returns too."""
_result, api_prompts, judge_prompts = await _simulate_with_judge()

assert "generated password, returns new ones each time." in " ".join(api_prompts[0].split())
assert "random or newly generated items by design may return new ones or repeat one." in " ".join(
judge_prompts[0].split()
)


async def test_the_simulator_and_its_judge_count_timestamps_from_the_unix_time():
_result, api_prompts, judge_prompts = await _simulate_with_judge()

assert "A tool that returns random or newly generated items by design" in " ".join(api_prompts[0].split())
assert "random or newly generated items by design returns new ones each time" in " ".join(judge_prompts[0].split())
assert "A Unix timestamp counts from the Unix time given there, 86,400 seconds a day." in " ".join(
api_prompts[0].split()
)
assert (
"Check a Unix timestamp by counting from the Unix time given there, 86,400 seconds a day, not by working "
"one out from the calendar." in " ".join(judge_prompts[0].split())
)


async def test_an_id_names_the_same_record_in_every_response():
_result, api_prompts, judge_prompts = await _simulate_with_judge()

assert (
"An ID an earlier response returned names the same record in every response: its name, title and other "
"fields stay as returned, and no other record takes that ID." in " ".join(api_prompts[0].split())
)
assert (
"It gives an ID an earlier response returned to a different record, or changes that record's name, title "
"or other fields that don't move with time." in " ".join(judge_prompts[0].split())
)


async def test_the_response_judge_checks_computed_values_but_not_fields_describing_the_response():
Expand Down
Loading