Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ The model is **Claude Haiku 5.5 on Vertex AI** (`claude-haiku-5-5`) by default.
| `evals/sync_cases.py` | Keeps `evals/.cases/` in line with the `agent_feedback` table: imports the promoted cases from the private case bucket, removes the retired ones, and with `--backfill` indexes cases that have no row ([Sync promoted cases from the bucket](#sync-promoted-cases-from-the-bucket)) |
| `evals/eligibility.py` | The model-free sweep of the catalogue's files of every Python library with a code profile (matplotlib, seaborn, plotnine, altair, pygal, lets-plot): eligible, coupled and blocked pairs, and why |
| `evals/scope.py`, `evals/scope.evalset.json` | The judge-only scope eval and its 219 synthetic cases: in-scope, off-topic and adversarial messages and notes pasted with a table, scored for adversarial recall and false refusals |
| `evals/prep.py`, `evals/fixtures/prep/` | The live prep eval and its 65 German and English cases (54 data requests, 10 of them `hard`, 11 data-code cases and 3 requests for example data, and 11 step-1 summaries): each through `/v1` in process with the real model, scored on the `prepare_data` operations and `generate_data` calls the root chose, on the size of the table each `run_data_code` call stored (a `refuse` case passes when nothing was stored), and on the shape of the root's reply ([Run the prep eval](#run-the-prep-eval)) |
| `evals/prep.py`, `evals/fixtures/prep/` | The live prep eval and its 71 German and English cases (59 data requests, 11 of them `hard`, 11 data-code cases and 3 requests for example data, and 12 step-1 summaries): each through `/v1` in process with the real model, scored on the `prepare_data` operations and `generate_data` calls the root chose, on the size of the table each `run_data_code` call stored (a `refuse` case passes when nothing was stored), and on the shape of the root's reply ([Run the prep eval](#run-the-prep-eval)) |
| `evals/fixtures/cases/` | The eval cases: 900 generated (`<spec>-<library>-<perturbation>`), eleven hand-written, among them `scatter-basic-matplotlib`, `bar-grouped-seaborn`, `scatter-basic-matplotlib-german-message` (a German follow-up that sets the session language) and four `*-combined-*` cases whose follow-up asks for two changes at once, each measured by the case's `checks`, and 23 stress cases (`stress-<name>`) ([fixtures README](evals/fixtures/README.md)) |
| `evals/baselines/` | Committed baseline reports, one per model ([baselines README](evals/baselines/README.md)) |

Expand Down Expand Up @@ -853,3 +853,5 @@ The two misses since the fix are model variance on the chain cases, not one caus
Since live test round 5 of 2026-10-15 a 51st case, `de-tagesmittel-pro-sensor` (tags `aggregate`, `mean`, `period`, `day`), asks for a daily mean per sensor of readings taken every 13 minutes, which the `day` period of `aggregate` answers ([#12200](https://github.com/MarkusNeusinger/anyplot/pull/12200)). Its run on 2026-10-15 with Claude Haiku 5.5 on `eu`, all 51 cases on the prompt that names the `hour`, `day` and `week` periods (root prompt hash `be470a15161bedb9`): 51 of 51, 11 of 11 data-code cases, 146 model calls, $0.137.

Since 2026-11-01 three cases (tag `example_data`) ask for example data without a table, each with the size it names, so the stored table's rows and columns are fixed: "Beispieldaten mit 4 Regionen und 3 Jahren" on `bar-grouped` (12 rows, 3 columns), "example data with 50 points" on `scatter-basic` (50 by 2) and 24 monthly values on `line-timeseries` (24 by 2). The root writes the code itself and calls `run_data_code` with `source_language` `example` (see [Two steps: data first, then the plot](../docs/concepts/agent-network.md#two-steps-data-first-then-the-plot)). Their first run, on 2026-11-01 with Claude Haiku 5.5 on `eu` (root prompt hash `1e36782cc5cef769`): 3 of 3, 10 model calls, $0.010, replies of 39 words at the median. The scope eval's seven `example_data` and `example_data_contrast` cases (five made-up requests in German, English and French in scope, two named companies' real figures `needs_data`) passed 7 of 7 the same day for $0.004.

Since 2026-11-04 six cases (tag `text`, and `bound_pair` for the summary) cover the text operations (see [Two steps: data first, then the plot](../docs/concepts/agent-network.md#two-steps-data-first-then-the-plot)): a `split` of an age range at the hyphen (German), an `extract` of a ticket number, a `strip` of a prefix, a `replace` of a hyphen by a space (German), a `hard` chain of a `strip` and a `split` that names every new column in the message, and the summary `en-summary-range-column` on `range-interval`. There the default bindings would put a count and an average on `min_value` and `max_value`, every minimum above its maximum, so the pair stays open and the session block lists the column of ranges as a `certain` split, which the summary owes (`score_conversions`).
107 changes: 96 additions & 11 deletions agents/anyplot/data/bindings.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,26 @@
`signs.py`) never takes a column with a negative value by default; it takes the next
candidate or stays unbound. Bound by hand, it is valid but warned about
(`BindingCheck.warnings`).

A lower and upper bound pair (`bounds.py`: `min_value` and `max_value`, `ci_lower` and
`ci_upper`, `low` and `high`) is never bound by default to two columns whose lower one starts
above the largest value of the upper one (`bounds.inverted`): every row's minimum would lie
above its maximum. Both roles stay open instead, so the conversion check for open roles runs
(`conversions.py`) and finds a column of ranges such as `10–25` to split. Bound so by hand,
the pair is the user's choice: valid, never blocked, kept through every change of the data
(`carry_bindings`), with one neutral warning line.
"""

import itertools
import re
from collections.abc import Collection, Mapping
from collections.abc import Collection, Mapping, Sequence
from dataclasses import dataclass, field
from difflib import SequenceMatcher

from core.spec_roles import DataRole, RoleKind

from ..schemas import Binding, ColumnDtype, ColumnProfile, DatasetProfile
from .bounds import bound_pairs, inverted, inverted_line, ordered
from .computed import match_score, window_line, window_rank
from .parse import value_like
from .shape import long_suggestion, takes_labels
Expand Down Expand Up @@ -127,7 +136,9 @@ class BindingCheck:
fits it has one (`signs.py`); and one per computed role bound to a column whose name
carries another window than the spec's (`computed.window_line`: `SMA 200` for a
20-period band); and one per role that takes labels bound to a measurement column of a
table that is wide for the spec (`sibling_line`: a heatmap's `y` on the numbers of `Mo`).
table that is wide for the spec (`sibling_line`: a heatmap's `y` on the numbers of `Mo`);
and one per lower and upper bound pair whose lower column starts above the largest value
of the upper one (`bounds.inverted_line`).
"""

complete: bool
Expand Down Expand Up @@ -252,9 +263,29 @@ def default_bindings(roles: list[DataRole], profile: DatasetProfile) -> list[Bin
it stands after the measure (`weight, X`, `_row_number_on_axis`).

Each column is bound at most once. Members of a family are numbered `<name>1`,
`<name>2`, ... in column order. Roles without kinds stay unbound.
`<name>2`, ... in column order. Roles without kinds stay unbound, and so do both roles of
a lower and upper bound pair whose columns would be inverted (`without_inverted`).
"""
return without_inverted(_with_row_numbers(roles, profile, set(label_siblings(roles, profile))), roles, profile)


def without_inverted(
bindings: list[Binding], roles: list[DataRole], profile: DatasetProfile, fixed: Collection[str] = ()
) -> list[Binding]:
"""`bindings` without both roles of every lower and upper bound pair whose columns are inverted
(`bounds.inverted`: the lower column starts above the largest value of the upper one).

A role in `fixed` keeps its binding (one the user made, or a version change kept): only the defaults
beside it give way, so the pair's other role stays open.
"""
return _with_row_numbers(roles, profile, set(label_siblings(roles, profile)))
columns = {column.name: column for column in profile.columns}
held = {binding.role: binding.column for binding in bindings}
dropped: set[str] = set()
for pair in bound_pairs(roles):
lower, upper = columns.get(held.get(pair.lower, "")), columns.get(held.get(pair.upper, ""))
if lower is not None and upper is not None and inverted(lower, upper):
dropped |= {pair.lower, pair.upper} - set(fixed)
return [binding for binding in bindings if binding.role not in dropped]


def _with_row_numbers(roles: list[DataRole], profile: DatasetProfile, siblings: set[str]) -> list[Binding]:
Expand Down Expand Up @@ -477,6 +508,7 @@ def carry_bindings(
previous: DatasetProfile | None = None,
converted: Mapping[str, str] | None = None,
row_numbers: Collection[str] = (),
splits: Sequence[Sequence[str]] = (),
) -> list[Binding]:
"""The bindings after the dataset changed under them (`prepare_data`, undo): the valid ones stay.

Expand All @@ -501,6 +533,12 @@ def carry_bindings(
A converted column that a role took leaves its source (`converted`, its values) out of the
optional roles: `Month` on `series` beside `Month (date)` on x would split the plot into
one series per row (owner's second opinion of 2026-11-02). A required role keeps it.

Two neighbouring number parts of one `split` (`splits`, `Prepared.splits`: `10–25`) fill an
open lower and upper bound pair, the first the lower role and the second the upper one. A
binding this function adds that would invert a pair (`bounds.inverted`) is left out, as in
`default_bindings`; a binding that stays is never touched, so a pair the user bound so on
purpose, high on the lower role, survives every change of the data.
"""
sources = converted or {}
kept = [binding for binding in bindings if not check_bindings([binding], roles, profile).errors]
Expand All @@ -512,7 +550,9 @@ def carry_bindings(
dated = _new_dates(freed, roles, fresh)
if dated:
current = [*freed, *dated]
current = [*current, *_new_numbers(current, roles, profile, sources)]
current = [*current, *_new_numbers(current, roles, profile, sources, splits)]
# Only what this function added gives way: a kept binding is the user's choice or an earlier one, never dropped.
current = without_inverted(current, roles, profile, {binding.role for binding in kept})
if current == kept:
return list(bindings)
return _without_sources(_in_spec_order(current, roles, profile), roles, sources)
Expand Down Expand Up @@ -622,6 +662,7 @@ def filled_bindings(kept: list[Binding], roles: list[DataRole], profile: Dataset
kept = [binding for binding in kept if not check_bindings([binding], roles, profile).errors]
used = {binding.column for binding in kept}
taken = {binding.role for binding in kept}
fixed = set(taken)
open_roles = [role for role in roles if role.variadic or role.name not in taken]
free = profile.model_copy(update={"columns": [column for column in profile.columns if column.name not in used]})
# The wide-table rule is judged on the whole table and spec, never on the reduced view.
Expand All @@ -632,7 +673,8 @@ def filled_bindings(kept: list[Binding], roles: list[DataRole], profile: Dataset
binding = Binding(role=role.member(number), column=binding.column)
taken.add(binding.role)
kept.append(binding)
return _in_spec_order(kept, roles, profile)
# A bound pair is judged with the kept bindings: a default upper bound below a kept lower one gives way.
return _in_spec_order(without_inverted(kept, roles, profile, fixed), roles, profile)


def _fresh_dates(kept: list[Binding], profile: DatasetProfile, previous: DatasetProfile | None) -> list[ColumnProfile]:
Expand Down Expand Up @@ -664,11 +706,20 @@ def _new_dates(kept: list[Binding], roles: list[DataRole], fresh: list[ColumnPro


def _new_numbers(
kept: list[Binding], roles: list[DataRole], profile: DatasetProfile, converted: Collection[str]
kept: list[Binding],
roles: list[DataRole],
profile: DatasetProfile,
converted: Mapping[str, str],
splits: Sequence[Sequence[str]] = (),
) -> list[Binding]:
"""Bindings of the open required roles that take numbers to the number columns a conversion made (`converted`).

A required family without a member takes the column as its first member (`y1`).
"""Bindings of the open required roles that take numbers to the number columns a conversion made (`converted`,
each with its source).

An open lower and upper bound pair takes two neighbouring parts of one `split` (`splits`), the first as the
lower bound, whatever order the spec lists the two roles in (`high` before `low`), and only when the first
neither starts nor ends above the second (`bounds.ordered`); otherwise the pair stays open. Two other columns
of one source (two `extract`s of it) never fill a pair together. A required family without a member takes
the column as its first member (`y1`).
"""
used = {binding.column for binding in kept}
fresh = [
Expand All @@ -680,6 +731,28 @@ def _new_numbers(
return []
bound = {role.name for binding in kept if (role := resolve_role(binding.role, roles)) is not None}
added: list[Binding] = []
by_name = {role.name: role for role in roles}
for pair in bound_pairs(roles):
if not all(_open_measure(by_name[name], bound) for name in (pair.lower, pair.upper)):
continue
columns = {column.name: column for column in fresh}
parts = next(
(
(columns[first], columns[second])
for split in splits
for first, second in itertools.pairwise(split)
if first in columns and second in columns
),
None,
)
if parts is None:
continue
# The parts of one cell belong together: the first must neither start nor end above the second
# (`bounds.ordered`), or the cells were written high to low (`22-18`) and the pair stays open, never swapped.
if ordered(*parts):
added += [Binding(role=pair.lower, column=parts[0].name), Binding(role=pair.upper, column=parts[1].name)]
Comment thread
MarkusNeusinger marked this conversation as resolved.
Resolved
bound |= {pair.lower, pair.upper}
fresh = [column for column in fresh if column not in parts]
for role in roles:
if not fresh or not role.required or role.computed or role.name in bound or "numeric" not in role.kinds:
continue
Expand All @@ -688,6 +761,11 @@ def _new_numbers(
return added


def _open_measure(role: DataRole, bound: Collection[str]) -> bool:
"""A required role that takes numbers, that the plot cannot compute and that no binding holds."""
return role.required and not role.computed and role.name not in bound and "numeric" in role.kinds


def _in_spec_order(bindings: list[Binding], roles: list[DataRole], profile: DatasetProfile) -> list[Binding]:
"""In the order the defaults list them: the spec's roles, a family's members by column position."""
role_order = {role.name: position for position, role in enumerate(roles)}
Expand All @@ -713,7 +791,8 @@ def check_bindings(bindings: list[Binding], roles: list[DataRole], profile: Data
a non-negative role to a column with a negative value (`signs.sign_line`), an
unbound non-negative role whose every fitting column that no other role holds has one,
and a role that takes labels bound to a measurement column of a wide table
(`label_siblings`, `sibling_line`).
(`label_siblings`, `sibling_line`), and a lower and upper bound pair whose lower column
starts above the largest value of the upper one (`bounds.inverted_line`).
"""
columns = {column.name: column for column in profile.columns}
siblings = label_siblings(roles, profile)
Expand All @@ -722,6 +801,7 @@ def check_bindings(bindings: list[Binding], roles: list[DataRole], profile: Data
seen_roles: set[str] = set()
seen_columns: set[str] = set()
bound: set[str] = set()
held: dict[str, ColumnProfile] = {} # the column of each valid binding, by its role's binding name
for binding in bindings:
role = resolve_role(binding.role, roles)
if role is None:
Expand All @@ -746,12 +826,17 @@ def check_bindings(bindings: list[Binding], roles: list[DataRole], profile: Data
errors.append(f"role '{binding.role}' needs {wanted} data, but column '{column.name}' is {column.dtype}")
continue
bound.add(role.name)
held[binding.role] = column
if column.name in siblings and takes_labels(role):
warnings.append(sibling_line(binding.role, column.name, siblings))
if role.non_negative and has_negative(column):
warnings.append(sign_line(binding.role, column))
if role.computed and (line := window_line(role.name, column.name)):
warnings.append(line) # nothing changes silently: `SMA 200` is no 20-period band
for pair in bound_pairs(roles):
lower, upper = held.get(pair.lower), held.get(pair.upper)
if lower is not None and upper is not None and inverted(lower, upper):
warnings.append(inverted_line(pair, lower, upper))
# An open non-negative role whose only fitting free columns are negative says why it stayed open;
# a column another role holds is no candidate, as in `default_bindings`.
for role in roles:
Expand Down
Loading
Loading