Skip to content

feat(agent): date_number turns a date column into numbers - #12558

Merged
MarkusNeusinger merged 7 commits into
mainfrom
feat/agent-date-number
Nov 5, 2026
Merged

feat(agent): date_number turns a date column into numbers#12558
MarkusNeusinger merged 7 commits into
mainfrom
feat/agent-date-number

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • date_number in the data step. prepare_data takes {"op": "date_number", "column", "unit": "year" | "days" | "timestamp" | "decimal_year", "name"?}. It adds a number column, appended last, beside a date column, which stays as it is. In the owner's test of 2026-11-04, a customers table on scatter-basic had Index on x and no column for y but Subscription Date, and "convert subscription date to integer" had no operation: set_type to number reads text cells, derive takes number columns, and make_date goes the other way.
    • days counts calendar dates from the column's earliest date (0).
    • timestamp is Unix seconds in UTC. A date counts from its midnight UTC, a time without an offset is read as UTC, and an offset is converted; the change line says which. Whole seconds stay whole numbers for every year from 1 to 9999. A fraction of a second is kept as decimals, and the change line counts the values a float64 rounds (more than 272 years from 1970).
    • year, days and decimal_year follow the column's own clock, as the aggregate buckets do.
    • A missing date stays empty, and the change line counts the dates and the empty cells. A column that is no date is a type_mismatch that tells the root to set_type it first.
    • Without name, the column is named after its source and the unit in the session language (apply_operations(language=...), passed by prepare_data and query_data).
    • The new column goes into Prepared.converted, so it fills an open role that takes numbers, and .undo() takes it back.
  • An offer, never a conversion by itself. When a required role that takes numbers only stays open and nothing else fills it, the session block lists a free date column as an offer (conversions.DateNumbers, never certain). The root states it in one sentence and makes it only on request; a request without a unit takes days. An offer never takes the place of a text-column conversion or of the row-number fallback.
  • The prep eval counts a closing invitation. closes_with_invitation flags a reply whose last sentence or clause asks the user to choose ("Choose …", "Pick one …", "tell me which …", "Wähle …"). Such a reply is a miss on the same terms as a closing question, and an open role that the block offered a date column for no longer excuses either. Three fixtures mirror the owner's case: English, German, and a summary that only offers.

Plan

N/A. The design is the owner's decision of 2026-11-04; the docs are in docs/concepts/agent-network.md ("Built since 2026-11-04 (dates as numbers)").

Texts to read

Change lines, one per unit

Example table: Index,Subscription Date with 2021-04-01, 2020-02-29 and one empty cell.

  • year: new column Subscription Date (year): 2 dates of Subscription Date read as calendar years, 1 cell without a date is empty; Subscription Date stays as it is
  • days: new column Subscription Date (days): 2 dates of Subscription Date read as whole days since 2020-02-29, the earliest date, which is 0, 1 cell without a date is empty; Subscription Date stays as it is
  • timestamp: new column Subscription Date (Unix time): 2 dates of Subscription Date read as Unix time in seconds (UTC), 1 cell without a date is empty; Subscription Date stays as it is, each date counted from its midnight UTC
  • decimal_year: new column Subscription Date (decimal year): 2 dates of Subscription Date read as decimal years (the year plus the part of it gone by), 1 cell without a date is empty; Subscription Date stays as it is

Variants of the change lines

  • days on a column at UTC+02:00: new column t (days): 2 dates of t read as whole days since 2025-01-01, the earliest date, which is 0, by the clock of UTC+02:00; t stays as it is
  • timestamp on a column with an offset: new column t (Unix time): 2 dates of t read as Unix time in seconds (UTC); t stays as it is, the moments converted to UTC from UTC+02:00
  • timestamp on a column the parser wrote in UTC: new column t (Unix time): 2 dates of t read as Unix time in seconds (UTC); t stays as it is, the moments as stored, in UTC
  • timestamp on times without an offset: new column t (Unix time): 2 dates of t read as Unix time in seconds (UTC); t stays as it is, times without a UTC offset read as UTC
  • timestamp with fractions of a second: new column t (Unix time): 2 dates of t read as Unix time in seconds (UTC); t stays as it is, times without a UTC offset read as UTC; fractions of a second kept as decimals, 1 value more than 272 years from 1970 rounded to the float's digits

Refusals

  • type_mismatch: column 'Index' is number; date_number needs a datetime column: set_type it to datetime first
  • invalid_operation: a column named 'Index' already exists
  • invalid_operation: column 'd' holds no date, so column 'd (year)' would be empty

Default column names

The server uses these when the root leaves name out (prepare.DATE_UNIT_WORDS; any other language takes English).

Language year days timestamp decimal_year
en Subscription Date (year) Subscription Date (days) Subscription Date (Unix time) Subscription Date (decimal year)
de Subscription Date (Jahr) Subscription Date (Tage) Subscription Date (Unix-Zeit) Subscription Date (Dezimaljahr)
fr (année) (jours) (temps Unix) (année décimale)
es (año) (días) (tiempo Unix) (año decimal)
it (anno) (giorni) (tempo Unix) (anno decimale)
pt (ano) (dias) (tempo Unix) (ano decimal)
nl (jaar) (dagen) (Unix-tijd) (decimaal jaar)
pl (rok) (dni) (czas uniksowy) (rok dziesiętny)

root.md, the op line (in the list under "Step 1: the data")

  • {"op": "date_number", "column": ..., "unit": "year" | "days" | "timestamp" | "decimal_year", "name": ...} (adds a number column computed from a date column, which stays: year is the calendar year; days the whole days since the column's earliest date, which is 0; timestamp the Unix time in seconds, UTC, a date without a time counted from its midnight and a time without an offset read as UTC; decimal_year the year plus the part of it gone by, 1 April 2021 is about 2021.25; a missing date stays empty; leave name out unless the user names the new column, the server names it after the column and the unit; a text column of dates takes a set_type to datetime first, in the same call; when the user asks for the dates as numbers without a unit ("convert the date to an integer", "als Zahl"), take days)

root.md, the offer bullet (after "Conversions for open roles")

  • A conversions line marked offer names a date column that date_number could turn into numbers for an open role that takes numbers. Unlike a certain line, which you make without asking, an offer waits for the user's request, because the unit is the user's choice: in the summary turn, and in a later turn about the plot or that role, say in one statement what is possible, such as "Subscription Date can become numbers for y: the days since the first date, the year, the Unix time or a decimal year.", and end the offer there: no question and no request to choose after it ("Choose …", "Pick one …", "Name the unit …", "Wähle …"), which are invitations too. When the user asks for it, call prepare_data with the line's operations, unit as the user asks; the server binds the new column to the role.

Session block: the conversions heading and the offer line

  • Conversions for open roles: {count}, columns that a set_type turns into what the role takes, or the row numbers when no column can, from a dry run on the data, and offers of a date column that date_number could turn into numbers (see 'Step 1: the data')

y: column 'Subscription Date' (dates): date_number would turn its 1000 dates into numbers in a new column; offer: the unit is the user's choice (the days since the earliest date, the year, the Unix time in seconds or a decimal year), so state it in one sentence, never as a question or a request to choose, and make it only when the user asks; operations with days: [{"op": "date_number", "column": "Subscription Date", "unit": "days"}]

The prepare_data tool description (its docstring, which the model reads) gains one clause:

date_number (column, unit year|days|timestamp|decimal_year, name; adds a number column from a date column: the calendar year, the whole days since the column's earliest date = 0, the Unix time in seconds in UTC, or the year plus the part of it gone by; name left out, the server names it after the column and the unit)

Test plan

  • uv run --extra dev ruff check . and ruff format --check .: clean.
  • uv run --extra typecheck mypy api core --pretty: clean.
  • uv run pytest tests/unit tests/integration -n auto: 26,853 passed, 209 skipped (Docker-only).
  • uv run python -m tools.changelog check --base origin/main: passes.
  • Gemini 3.8 Flash check on 4f1166d91: no correctness, regression or safety finding. Its three test gaps are added in d87f8b1bb: days and decimal year on an offset timestamp, complete bindings left untouched, and no offer for an all-missing column.
  • Unit tests:
    • each unit; a time of day (calendar dates for days, seconds for timestamp); offsets converted for timestamp and kept as the clock for the other units;
    • a column the parser wrote in UTC; fractions and the rounding count; years 1 and 9999;
    • missing cells; the original column kept; a text or number column refused; a compact YYYYMM column after its set_type in the same call; a column with no date left refused;
    • default names per language and their numbering; an explicit name; validation through the models and ToolSafety (unit, name length, extra key, a link in column).
  • Binding and undo:
    • carry_bindings fills the open y, and complete bindings stay as they are;
    • push_version and undo_version restore the version before, with y open again;
    • an end-to-end service-flow turn in a German session makes Subscription Date (Tage), binds y, and undo takes it back;
    • query_data names the column in the session language.
  • Offer line: it appears in the owner's case; its operations fill the role; a text conversion and the row-number fallback come first; no offer for a role that takes dates, for a bound date column, or for an all-missing one.
  • Mutation run: an in-memory runner recompiled one function per mutant, with no repo file touched. All 26 mutants turned a test red:
    • units: year, days origin, calendar days, leap year;
    • timestamp: the offset, its notice, fractions, the rounding count; the column's clock for the year;
    • missing cells, the empty count, the source column kept, the type check;
    • the converted binding and the session-language name;
    • offer: yields to other lines, numbers-only roles, never certain;
    • undo, and the moved name helper of the existing ops;
    • Flash's three: _wall in UTC, a bound role reassigned, an all-missing column offered;
    • the eval check: an invitation never a miss, an offer still blocking, last sentence only.
  • False-positive scan of closes_with_invitation over the 40 replies captured in the live runs: it flags exactly the 3 known invitations and none of the other 37, which span 15 cases in English and German.
  • Live prep eval, Claude Haiku 5.5 on Vertex eu, in process with ADC:
    • First wordings: 3 of 3 cases, 3 of 3 again, and 15 of 15 together with all 13 summaries (38 model calls, $0.038).
    • On the merged prompt: 3 of 3 for $0.018. In 2 of the 5 summary-offer runs so far, the reply closed with an invitation, such as "Choose the y column in the data panel, or name the unit you want."
    • Final wording (root prompt hash 30f3d5615c83b581): the summary offer matched in 6 of 6 runs, each closing with a statement such as "Subscription Date can become numbers for y: the days since the first date, the year, the Unix time or a decimal year." Both message cases matched with days and no name; the outage of one run (guard_unavailable) was re-run. About $0.024.
    • Replies on the final wording. English: "I converted Subscription Date into whole days since its earliest date, 2020-03-25 (day 0), for all 8 rows. The new column, Subscription Date (days), is now bound to y, and the original date column stays as it is. Check the preview in the data panel, then press .create_plot() beside the message box. .undo() brings back the earlier data." German: "Ich habe das Abodatum in eine ganze Zahl umgewandelt: die Tage seit dem frühesten Datum (25.03.2020 = 0), für alle 8 Zeilen. Die neue Spalte heißt „Abodatum (Tage)" und ist jetzt die y-Achse; das Originaldatum bleibt erhalten. Prüfe die Vorschau im Datenbereich und drücke dann .create_plot()."
    • The root never asked which unit. All live runs together cost about $0.15.

Known issues not fixed here

  • The root sometimes calls its own required row-number conversion a mistake. In de-summary-zeilennummer it wrote "ohne deinen Auftrag … Das war mein Fehler", including on origin/main's prompt (2 of 3 runs, one of them a miss). It predates this change and needs its own fix in root.md.
  • A German Nr column of 1 to n is not a row-number name (bindings.ROW_NUMBER_NAMES). It sits on x, and the row-number fallback then moves it to y and puts new row numbers on x, which draws a diagonal.

https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8

MarkusNeusinger and others added 4 commits November 4, 2026 22:48
@MarkusNeusinger @claude
feat(agent): date_number turns a date column into numbers
bf29582 
The owner's test of 2026-11-04: a customers table on scatter-basic had
Index on x, nothing for y but its subscription dates, and "convert
subscription date to integer" had no operation (set_type to number reads
text cells, derive takes number columns, make_date goes the other way).

- prepare.py: `date_number` (column, unit year|days|timestamp|decimal_year,
  name) adds a number column beside the date column. Days count calendar
  dates from the earliest (0); the timestamp is Unix seconds in UTC, a
  date at its midnight UTC, a time without an offset read as UTC, an
  offset converted, fractions kept (with a count of values a float64
  rounds); year, days and decimal year keep the column's clock. Missing
  dates stay empty and are counted; a non-date column is a type_mismatch
  with a set_type hint. Without `name` the column is named after its
  source and unit in the session language (`apply_operations(language=)`).
  The new column goes into `Prepared.converted`, so it fills an open
  number role; undo works as for every version.
- conversions.py: an open required role that takes numbers only, which no
  conversion or row number fills, gets an offer line for a free date
  column (`DateNumbers`, never certain). `suffixed_name` moved to
  prepare.py for both.
- root.md: the op and the offer bullet (one statement, never a question
  or an invitation; a request without a unit takes days).
- session/query tools pass the session language; telemetry PrepOp, eval
  describe, three prep fixtures (en, de, a summary that only offers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
docs(agent): the offer contrasts with a certain line; prep eval notes
713ddc3 
root.md: an offer waits for the user's request, unlike a certain line,
which the root makes without asking (the earlier "never make it unasked"
could read as covering every conversion). agents/README.md: the three
date_number prep cases and their live runs on Claude Haiku 5.5 (eu).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
Merge origin/main (#12557) into feat/agent-date-number
4f1166d 
The one conflict was the prep fixture count in test_prep.py: 71 cases on
the old main, plus the four plot-type cases of #12557 and the three
date_number cases of this branch, is 78.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
test(agent): Flash's three gaps; the offer states, the eval counts in…
d87f8b1 
…vitations

Flash on 4f1166d (no correctness, regression or safety finding) named
three test gaps, each now with a test and a mutant it catches:
- days and decimal_year on 2025-01-01T00:30+02:00 keep the column's clock
  (break: `_wall` converting to UTC first);
- a date_number on data whose x and y are bound leaves the bindings as they
  are (break: `_new_numbers` taking a bound role);
- an all-missing date column is no offer (break: the `missing < rows`
  clause dropped).

root.md: the offer bullet ends the offer at the statement, with the
imperative closers named as invitations; the conversions line says "never
as a question or a request to choose". 2 of 5 earlier summary replies had
closed with "Choose the y column in the data panel, or name the unit you
want."; 0 of 6 on the final wording.

prep eval: `closes_with_invitation` (a narrow list of choose/tell verbs at
the start of the last sentence or clause, never the .confirm() closer)
makes a reply miss on the same terms as a closing question, and an open
role the block offered a date column for (`offered`) excuses neither. It
flags the three live invitations and none of the 37 other replies of this
session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Copilot AI lite review requested due to automatic review settings November 4, 2026 22:31
Copilot AI reviewed Nov 4, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two active nit findings remain: the summary-fixture inventory is outdated, and timestamp rounding detection can report exact far-future whole-second values as rounded.

2 open issues
What changed in this PR

This pull request adds date_number conversion for turning date columns into years, elapsed days, Unix timestamps, or decimal years while preserving the original column. It also adds localized naming, binding and undo support, date-number offers for open numeric roles, and expanded evaluation coverage.

Changes:

  • Adds and validates date-to-number preparation and query operations, including timezone handling, missing values, precision reporting, and localized names.
  • Adds date conversion offers, binding carryover, undo support, telemetry, prompts, and tool documentation.
  • Extends unit, integration, evaluation, fixture, documentation, and changelog coverage, including closing-invitation detection.

Review status: This pull request requires changes.

File Summary Review notes
tests/​unit/​agents/​runtime/​test_service_flow.py Covers the German end-to-end offer, conversion, binding, and undo flow. —
tests/​unit/​agents/​runtime/​test_plugins.py Validates date_number tool argument safety. —
tests/​unit/​agents/​runtime/​test_dataset_versions.py Verifies converted bindings and version undo behavior. —
tests/​unit/​agents/​runtime/​test_data_questions.py Verifies localized generated-column names in queries. —
tests/​unit/​agents/​evals/​test_prep.py Tests invitation detection, reply scoring, and offer-aware evaluation. —
tests/​unit/​agents/​data/​test_prepare.py Covers date-number units, timezone behavior, precision, validation, naming, and edge cases. —
tests/​unit/​agents/​data/​test_conversions.py Covers date-number offers, precedence, role compatibility, and fallback behavior. —
docs/​concepts/​agent-network.md Documents date-number conversion and offer semantics. —
changelog.d/​agent-date-number.md Records the new feature and evaluation changes. —
agents/​README.md Documents preparation evaluation behavior and date-number fixtures. Nit: Update the summary-fixture inventory to include the new ninth summary fixture and reflect the 13-case set.
agents/​evals/​prep.py Detects closing invitations and accounts for date-number offers in reply scoring. —
agents/​evals/​fixtures/​prep/​en-summary-date-offer.json Adds the English summary-only date-number offer fixture. —
agents/​evals/​fixtures/​prep/​en-date-as-integer.json Adds the English date-to-integer fixture. —
agents/​evals/​fixtures/​prep/​de-datum-als-ganzzahl.json Adds the German date-to-integer fixture. —
agents/​anyplot/​tools/​session.py Exposes date_number and propagates session language. —
agents/​anyplot/​telemetry.py Registers the new preparation operation. —
agents/​anyplot/​prompts/​root.md Documents the operation, default unit, and offer behavior. —
agents/​anyplot/​data/​query.py Propagates session language for query-generated names. —
agents/​anyplot/​data/​prepare.py Implements date-number conversion, date arithmetic, timestamps, naming, and precision reporting. Nit: Count only timestamp values whose exact microsecond values are actually rounded by float64, rather than all values beyond the magnitude threshold when any fractional timestamp exists.
agents/​anyplot/​data/​conversions.py Adds date-number offers for open numeric roles and integrates them into conversion selection. Moderate: Restrict date-number offers to roles whose accepted kinds are exactly numeric; roles accepting both text and numeric values must not receive this offer.
agents/​anyplot/​agent.py Includes date-number offers in the session conversion context. —

Review effort: Lite

Comment thread agents/README.md Resolved
Comment thread agents/anyplot/data/prepare.py Resolved
@MarkusNeusinger @claude
fix(agent): count only timestamps whose text loses a digit; README in…
2e2a1ca 
…ventory

Copilot on #12558, two low findings, both applied:
- prepare.py: the rounding count of a timestamp column with fractions went by
  magnitude (2**33 seconds) and so counted a far whole second, which a
  float64 holds exactly (2300-01-01 is 10413792000.0). It now compares the
  text data.csv writes (the float's shortest repr) with the exact
  microseconds. New test for the far whole second; both directions have a
  mutant that turns a test red.
- agents/README.md: "the eight summary cases" had gone stale (13 now); it
  names the first eight and points to the paragraphs that add the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@codecov

codecov Bot commented Nov 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@MarkusNeusinger @claude
test(agent): a whitespace-only reply closes with no invitation
8b7ba8c 
Codecov on #12558: the empty-reply branch of `closes_with_invitation` was the
one line of the patch without a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger
MarkusNeusinger merged commit aa7a5b0 into main Nov 5, 2026
16 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the feat/agent-date-number branch November 5, 2026 07:21
MarkusNeusinger added a commit that referenced this pull request Nov 5, 2026
@MarkusNeusinger @claude
Merge origin/main into feat/agent-chat-one-field (date_number, #12558)
eec6474 
Conflicts resolved: prepare.py's _Carried keeps main's `language` beside
the branch's kept cells and row flag; prep.py's case runner lists the
conversions once (main's _listed_conversions) and owes their certain
lines on every summary turn, the note's included; the reply-miss docs
and the gate's text name the invitation, the long reply and the missing
row count; test_prep.py counts 84 cases and keeps both new test groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MarkusNeusinger