Skip to content

fix(engine): let users write what they ask about, and keep the app's context line out of their view - #165

Merged
3mei merged 2 commits into
mainfrom
yev/fix/user-language-and-context
Nov 16, 2026
Merged

3mei merged 2 commits into
mainfrom
yev/fix/user-language-and-context

Conversation

@3mei

@3mei 3mei commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Related issue

None.

What changed and why

Two follow-ups to how the simulated user's messages are instructed and shown.

  • A user turn may hold the text it asks about in another language. Every locale's simulated user is told to write in the locale's language, English locales are told not to write phrases or sentences in another language, and every locale's user-turn gate fails a turn that does. A user asking for a translation, or practising a language and asking for corrections, has to write the phrase in question, so general_open_ended's language_translation topic and general_educational's language-learning subject drew retries, language violations and gate exhaustion. Every language rule now leaves such text alone: the user's instruction and per-turn requirement say that text it asks about, such as a phrase to translate or a sentence to have corrected, may stay in its own language, and every gate clause, romanized ones included, holds only the words around it to the locale's language. A turn written in another language still fails.
  • The simulated user no longer sees the app's context line as its own. A probe's user_context line, such as tool_calling's [2026-11-06 · Austin, Texas, USA], opens the first message the assistant sees, and the stored transcript keeps it. The follow-up prompt and the follow-up gate read their history from that transcript, so the user saw a line it never typed at the top of its own opening. Both now read the first user message as the user wrote it; the transcript, the assistant and the assistant judge keep the line.

docs/engine/AUTHORING_A_PROBE.md describes the exception in its account of language guidance.

How it was verified

New tests fail on the code before the change and pass after it: every locale's instruction, per-turn requirement and gate clause allow the text a turn asks about (test_user_language.py, 33 locales), and with a context line set, neither the user's follow-up prompts nor the user-turn gates contain it (test_simulation.py).

On each commit's own tree: make check-all, the pre-commit hooks on its files and pytest --collect-only (5233, then 5234 tests collected).

At the head, on its exact tree in a fresh environment:

$ make check-all
471 files already formatted
$ make test
5234 passed in 92.11s (0:01:32)
$ make smoke
To run a real simulation:
$ make check-license-headers
   ❌ Need update: 0
   ✅ Up to date: 473
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs

Live, against a custom provider config, on these two commits over the previous main (the identity probe changes that landed since touch none of their files): the end-to-end health check with its default probe and locale, and with the tool_calling probe in English, the evaluator answering every request in both:

$ make health-check PROVIDER_CONFIG=<custom config>
✓ simulate, eval and report all completed
$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=en_US
✓ simulate, eval and report all completed

Results impact

Every probe's simulated user hears the new allowance in its instructions, and every locale's user-turn gate carries it, so generated user turns can change in any probe; conversations on translation or language practice should see fewer language retries and failed openings. In conversations from a probe with a user_context line (tool_calling), the simulated user's follow-ups are written without that line in view. No probe's prompt version changes.

3mei added 2 commits November 16, 2026 13:22
…anguage

Every locale's simulated user is told to write in the locale's language,
and English locales are told not to write phrases or sentences in
another language; every locale's user-turn gate fails such a turn as a
language violation. A user asking for a translation, or practising a
language and asking for corrections, has to write the phrase in
question, so general_open_ended's language_translation topic and
general_educational's language-learning subject drew retries,
n_user_language_violations and gate exhaustion.

Every language rule now leaves that text alone. The user's instruction
and its per-turn requirement say that text it asks about, such as a
phrase to translate or a sentence to have corrected, may stay in its own
language, and every gate clause, romanized ones included, holds only the
words around it to the locale's language. A turn written in another
language still fails.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A probe's user_context line opens the first user message the assistant
sees, as the assistant's app would attach it, and the stored transcript
keeps it. The simulated user's follow-up prompt and the follow-up gate
read their history from that transcript, so the user saw its own
opening as "[2026-11-06 · Austin, Texas, USA]" followed by its text, a
line it never typed.

Both now read the first user message as the user wrote it. The stored
transcript, the assistant and the assistant judge keep the line.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner November 16, 2026 18:23
@3mei
3mei merged commit 207ae01 into main Nov 16, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant