Skip to content
4 changes: 3 additions & 1 deletion agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -822,7 +822,7 @@ The prep eval (`evals/prep.py`) measures the one non-deterministic part of the s

A case passes on exact operation match, order-insensitive, when every `prepare_data` and `generate_data` call of the turn succeeded (the operations apply in order, so a list the data refuses, such as a column named before its rename, is a miss however it reads, and so is a formula the server cannot compute): the operations of every call, taken together, equal the expected ones after both went through the `Operation` models (a default the model left out and one it wrote are the same; a `set_type` `decimal` equal to the dataset's own counts as left out; the column lists of `drop_columns` and `pivot_longer` and the conditions of `drop_rows` are sets). A `generate_data` call is one expected item `{"op": "generate_data", "variables": [...], "series": [...]}`, and two calls match when they compute the same table: the same names, points and values, so an endpoint written `-3.0` or `-3`, `tau` or `2*pi`, and an expression written `x*x` or `x**2` are the same choice (`comparison_key`). The report lists the call after the `Generate` model with its defaults left out (a variable named `x`, 200 points), the whitespace of every expression and endpoint removed, the series as a set and the variables in order. A case that expects no operation passes when the root made no data call, whether the scope judge refused the message or the root answered in words; the report lists every tool call and the refusal code. A reply in which the root talks about its own instructions (ADK's instruction markers, a system instruction or the session block) is a miss whatever the data calls did, and the summary counts such cases under `instruction_talk`. The gate is 80 % exact matches (`--min-exact`); a case whose turn ended in an `error` event counts in no rate, and more than 10 % of them (`--max-unavailable`) exit 4.

Since 2026-11-01 (owner decision: "short and to the point, no waffle, follow-up questions only when they really matter") the eval also measures every reply of the root: its words, its questions, and whether it closes with a question (`measure_reply`). A reply is a reply miss when it closes with a question the turn was not blocked for, or when it runs over 100 words (`MAX_REPLY_WORDS`, about the five sentences `root.md` allows at most). A turn is blocked when its case carries the tag `ask_back` (a request that can mean two different changes, or dates whose day and month order reads both ways), when a data call of the turn failed, or when the turn left a required role of the spec without a column (`roles_open`), the case `root.md` names for a question. The reply gate is at most 10 % reply misses among the answered cases with a reply (`--max-reply-misses`); the summary reports the median and the longest reply in words, the questions per reply and the closing questions. A case can send the page's step-1 summary action instead of a message (`"action": "summarize_data"`, with a dataset); the eight summary cases (tag `summary`) are a German pie of pets, a German wide table of expenses that `.to_long()` fits, an English daily series, English slash dates whose order reads both ways (it may close with its question), and, since the live test of 2026-11-01, the owner's RTX 4090 price table (`Nov 2026` to `Dec 2025`, newest first, a change column with `−` signs) and its German twin (`Okt. 2026`, prices in €), whose only expected operation is the `sort` of the month column, an English column of `8 Nov 2026`-style dates that converts for certain (the `set_type` into `Date (date)` and the sort, `data/conversions.py`), and German dates with two-digit years (`03.11.26`), an uncertain conversion that may close with its question and expects none. In the user test of 2026-11-01, 10 of 11 summaries closed with a question that changed nothing. The reply text never enters the report.
Since 2026-11-01 (owner decision: "short and to the point, no waffle, follow-up questions only when they really matter") the eval also measures every reply of the root: its words, its questions, and whether it closes with a question (`measure_reply`). A reply is a reply miss when it closes with a question the turn was not blocked for, or when it runs over 100 words (`MAX_REPLY_WORDS`, about the five sentences `root.md` allows at most). A turn is blocked when its case carries the tag `ask_back` (a request that can mean two different changes, or dates whose day and month order reads both ways), when a data call of the turn failed, or when the turn left a required role of the spec without a column (`roles_open`), the case `root.md` names for a question, unless the session block offered a date column for that role (`offered`), which the reply states instead. Since 2026-11-04 a closing invitation to choose is a reply miss on the same terms (`closes_with_invitation`: a last sentence or clause such as "Choose …", "Pick one …", "tell me which …" or "Wähle …", never the instruction to press `.confirm()`); the summary counts both. The reply gate is at most 10 % reply misses among the answered cases with a reply (`--max-reply-misses`); the summary reports the median and the longest reply in words, the questions per reply and the closing questions. A case can send the page's step-1 summary action instead of a message (`"action": "summarize_data"`, with a dataset); the first eight of the 13 summary cases (tag `summary`; the later ones are named in the paragraphs below) are a German pie of pets, a German wide table of expenses that `.to_long()` fits, an English daily series, English slash dates whose order reads both ways (it may close with its question), and, since the live test of 2026-11-01, the owner's RTX 4090 price table (`Nov 2026` to `Dec 2025`, newest first, a change column with `−` signs) and its German twin (`Okt. 2026`, prices in €), whose only expected operation is the `sort` of the month column, an English column of `8 Nov 2026`-style dates that converts for certain (the `set_type` into `Date (date)` and the sort, `data/conversions.py`), and German dates with two-digit years (`03.11.26`), an uncertain conversion that may close with its question and expects none. In the user test of 2026-11-01, 10 of 11 summaries closed with a question that changed nothing. The reply text never enters the report.

Since 2026-11-02 a summary case is also checked for the conversions it owes (`score_conversions`): before the turn the run computes in code what the session block will list for the open roles (`conversions.candidates`), and after it each `certain` line must be among the root's operations. A root that skips one is the original failure of the live test of 2026-11-01, so the case is a miss whatever its fixture expected, and any skip fails the run. The summary counts `conversions_owed` and `conversions_made`, which is how often the root applies a `certain` conversion. Three summary cases joined: a list of numbers on `line-basic` in English and German, which gets its row numbers (`row_number`), and a `Year` column for a date role, read as years.

Expand Down Expand Up @@ -855,3 +855,5 @@ Since live test round 5 of 2026-10-15 a 51st case, `de-tagesmittel-pro-sensor` (
Since 2026-11-01 three cases (tag `example_data`) ask for example data without a table, each with the size it names, so the stored table's rows and columns are fixed: "Beispieldaten mit 4 Regionen und 3 Jahren" on `bar-grouped` (12 rows, 3 columns), "example data with 50 points" on `scatter-basic` (50 by 2) and 24 monthly values on `line-timeseries` (24 by 2). The root writes the code itself and calls `run_data_code` with `source_language` `example` (see [Two steps: data first, then the plot](../docs/concepts/agent-network.md#two-steps-data-first-then-the-plot)). Their first run, on 2026-11-01 with Claude Haiku 5.5 on `eu` (root prompt hash `1e36782cc5cef769`): 3 of 3, 10 model calls, $0.010, replies of 39 words at the median. The scope eval's seven `example_data` and `example_data_contrast` cases (five made-up requests in German, English and French in scope, two named companies' real figures `needs_data`) passed 7 of 7 the same day for $0.004.

Since 2026-11-04 six cases (tag `text`, and `bound_pair` for the summary) cover the text operations (see [Two steps: data first, then the plot](../docs/concepts/agent-network.md#two-steps-data-first-then-the-plot)): a `split` of an age range at the hyphen (German), an `extract` of a ticket number, a `strip` of a prefix, a `replace` of a hyphen by a space (German), a `hard` chain of a `strip` and a `split` that names every new column in the message, and the summary `en-summary-range-column` on `range-interval`. There the default bindings would put a count and an average on `min_value` and `max_value`, every minimum above its maximum, so the pair stays open and the session block lists the column of ranges as a `certain` split, which the summary owes (`score_conversions`).

Since 2026-11-04 three cases (tag `date_number`) mirror the owner's customers table on `scatter-basic`, `Index` on x and nothing for y but the subscription dates: "convert subscription date to integer" and its German variant expect one `date_number` with `days`, the plain reading of a request without a unit, and no `name`, which the server gives in the session language; the summary `en-summary-date-offer` expects no operation, because the session block lists the dates only as an offer. Live runs that day with Claude Haiku 5.5 on `eu` matched every time. On the earlier wordings 2 of 5 summary replies closed with an invitation to choose ("Choose the y column in the data panel, or name the unit you want."), which `closes_with_invitation` now counts as a reply miss; on the final wording (root prompt hash `30f3d5615c83b581`) the summary matched in 6 of 6 runs, each closing with the offer as a statement, and both message cases matched, for $0.024 in all. One run of an earlier wording with all 13 summaries matched 15 of 15 (38 model calls, $0.038).
Comment thread
MarkusNeusinger marked this conversation as resolved.
Resolved
3 changes: 2 additions & 1 deletion agents/anyplot/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,8 @@
"""The session block's line when the data came from a dropped file: a fixed text, never the name itself."""
CONVERSIONS_HEADING = (
"Conversions for open roles: {count}, columns that a set_type turns into what the role takes, or the row "
"numbers when no column can, from a dry run on the data (see 'Step 1: the data')"
"numbers when no column can, from a dry run on the data, and offers of a date column that date_number could turn "
"into numbers (see 'Step 1: the data')"
)
"""The session block's line over the conversion candidates of the roles left open (`data/conversions.py`), live test
of 2026-11-01: the root asked the user for a date column that a conversion of the `Month` text would have made."""
Expand Down
116 changes: 96 additions & 20 deletions agents/anyplot/data/conversions.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,14 @@
root offers it as its one question. A row-number column the paste already has is reused, and
nothing here runs for `/dataset/copy`, which copies confirmed data byte for byte.

A date column is never a number candidate, but a required role that takes numbers only (`bindings.is_measure`),
left open by every line above, gets an offer when a date column is free (`DateNumbers`, owner decision of
2026-11-04: a customers table on `scatter-basic` had `y` open beside its `Subscription Date`). `date_number` would
turn the dates into numbers, and the unit is the user's choice (the year, the days since the earliest date, the
Unix time or a decimal year), so the line is neither `certain` nor a question: the root names the offer in one
statement and makes it only on request. Its operations carry `days`, the plain reading of a request for "a
number" without a unit (`root.md`), and no name, which the server gives in the session language.

The new column is named after its source with the type in the session language
(`Month (date)`, `Monat (Datum)`, `TYPE_WORDS`): the root takes the operations as given, so
the conversion is the same in every run. Each dry run is made once per dataset version
Expand All @@ -69,11 +77,20 @@
from core.spec_roles import DataRole

from ..language import DEFAULT_LANGUAGE, canonical
from ..schemas import MAX_COLUMN_NAME_CHARS, Binding, ColumnProfile, DatasetProfile
from .bindings import RowSlot, axis_slot, check_bindings, code_like, row_number_columns, row_slot
from ..schemas import Binding, ColumnProfile, DatasetProfile
from .bindings import (
RowSlot,
axis_slot,
check_bindings,
code_like,
is_measure,
resolve_role,
row_number_columns,
row_slot,
)
from .bounds import BoundPair, bound_pairs
from .parse import ParsedDataset, id_like, span_like
from .prepare import ROW_NUMBER_NAME, DryRun, RangeRun, dry_run, range_run, read_frame, year_run
from .prepare import ROW_NUMBER_NAME, DryRun, RangeRun, dry_run, range_run, read_frame, suffixed_name, year_run
from .temporal import year_score


Expand Down Expand Up @@ -235,6 +252,42 @@ def line(self) -> str:
return f"{line}; operations: {json.dumps(self.operations(), ensure_ascii=False)}"


@dataclass(frozen=True)
class DateNumbers:
"""A free date column that `date_number` would turn into numbers for an open required role that takes numbers only.

An offer, never made unasked: the unit is the user's choice, so `certain` is False, and the root names the offer
in one statement instead of asking (`root.md`, "Conversions for open roles").
"""

role: str
column: str
dates: int
"""The column's cells that hold a date."""

@property
def certain(self) -> bool:
return False

@property
def outcome(self) -> Outcome:
return "number"

def operations(self) -> list[dict[str, Any]]:
"""The `date_number` with the plain reading of "as a number", `days`; the server names the new column."""
return [{"op": "date_number", "column": self.column, "unit": "days"}]

def line(self) -> str:
"""The session block's line: the role, the column, the count, why it is an offer, and the operations."""
line = (
f"{self.role}: column '{self.column}' (dates): date_number would turn its {self.dates} dates into numbers "
"in a new column; offer: the unit is the user's choice (the days since the earliest date, the year, the "
"Unix time in seconds or a decimal year), so state it in one sentence, never as a question or a request "
"to choose, and make it only when the user asks"
)
return f"{line}; operations with days: {json.dumps(self.operations(), ensure_ascii=False)}"


def stays_text(column: ColumnProfile) -> bool:
"""A column no conversion touches: ids, codes or spans of years, by its name or its frequent cells."""
return id_like(column.name) or span_like(column.name) or code_like(column)
Expand All @@ -249,15 +302,17 @@ def candidates(
language: str = DEFAULT_LANGUAGE,
keep_text: Collection[str] = (),
cache: MutableMapping[tuple[str, str], DryRun | RangeRun | None] | None = None,
) -> list[Candidate | RangeSplit | RowNumbers]:
) -> list[Candidate | RangeSplit | RowNumbers | DateNumbers]:
"""The conversions that would fill the required roles `bindings` leave open, at most `MAX_CANDIDATES`.

First an open lower and upper bound pair takes the first free column of ranges (`RangeSplit`);
then the roles in spec order, each taking the first free column (in column order) whose dry
run reads it as the role's type; a column serves one role. `keep_text` names the
columns kept text on request, `cache` holds the dry runs by column and type
(`StoredDataset.dry_runs`; `range` for the ranges); `parsed` is read into a frame only when a
dry run is due. With no conversion at all, the row numbers (`RowNumbers`).
dry run is due. With no conversion at all, the row numbers (`RowNumbers`). Last, an offer of a
free date column for each open role that takes numbers only and that none of these fills
(`DateNumbers`).
"""
missing = set(check_bindings(bindings, roles, profile).missing_roles)
if not missing:
Expand Down Expand Up @@ -335,7 +390,39 @@ def ranges(column: str) -> RangeRun | None:
on_axis = slot is not None and role.name == slot.role.name
found.append(Candidate(role.name, dry, name, moved if on_axis else None))
break
return found or _row_numbers(bindings, roles, profile, used)
filled = found or _row_numbers(bindings, roles, profile, used)
return [*filled, *_date_numbers(roles, profile, missing, used, filled)][:MAX_CANDIDATES]


def _date_numbers(
roles: list[DataRole],
profile: DatasetProfile,
missing: Collection[str],
used: Collection[str],
filled: list[Candidate | RangeSplit | RowNumbers],
) -> list[Candidate | RangeSplit | RowNumbers | DateNumbers]:
"""An offer per open required role that takes numbers only and that no line of `filled` fills, in spec order, each
with the next free date column that holds a date (in column order)."""
named: set[str] = set()
for line in filled:
if isinstance(line, RangeSplit):
named |= {line.pair.lower, line.pair.upper}
continue
moved = line.slot.moved if isinstance(line, RowNumbers) else line.moved
named |= {line.role, *([moved.role] if moved is not None else [])}
covered = {role.name for name in named if (role := resolve_role(name, roles)) is not None}
dates = [
column
for column in profile.columns
if column.dtype == "datetime" and column.name not in used and column.missing < profile.rows
]
offers: list[Candidate | RangeSplit | RowNumbers | DateNumbers] = []
for role in roles:
if not dates or role.name not in missing or role.name in covered or role.computed or not is_measure(role):
continue
column = dates.pop(0)
offers.append(DateNumbers(role.name, column.name, profile.rows - column.missing))
return offers


def _row_numbers(
Expand All @@ -356,22 +443,11 @@ def _row_numbers(

def new_name(column: str, target: Target, language: str, taken: Collection[str]) -> str:
"""`<column> (<type word>)` within the column-name cap, `_2`, `_3`, ... before the bracket when it is taken."""
return _with_word(column, TYPE_WORDS.get(canonical(language), TYPE_WORDS[DEFAULT_LANGUAGE])[target], taken)
return suffixed_name(column, TYPE_WORDS.get(canonical(language), TYPE_WORDS[DEFAULT_LANGUAGE])[target], taken)


def range_names(column: str, language: str, taken: Collection[str]) -> tuple[str, str]:
"""`<column> (min)` and `<column> (max)` in the session language (`RANGE_WORDS`), as `new_name` names them."""
low, high = RANGE_WORDS.get(canonical(language), RANGE_WORDS[DEFAULT_LANGUAGE])
first = _with_word(column, low, taken)
return first, _with_word(column, high, {*taken, first})


def _with_word(column: str, word: str, taken: Collection[str]) -> str:
suffix = f" ({word})"
stem = column[: MAX_COLUMN_NAME_CHARS - len(suffix)].rstrip()
name, number = stem + suffix, 2
while name in taken:
tail = f"_{number}"
name = column[: MAX_COLUMN_NAME_CHARS - len(suffix) - len(tail)].rstrip() + tail + suffix
number += 1
return name
first = suffixed_name(column, low, taken)
return first, suffixed_name(column, high, {*taken, first})
Loading
Loading