Skip to content

fix(engine): answer after the tool-call limit, and don't stop on a request - #174

Merged
3mei merged 2 commits into
mainfrom
yev/fix/tool-cap-reply-and-early-stop
Nov 20, 2026
Merged

3mei merged 2 commits into
mainfrom
yev/fix/tool-cap-reply-and-early-stop

Conversation

@3mei

@3mei 3mei commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Related issue

None.

What changed and why

Two fixes to the shared conversation loop, both about a turn ending without the answer the user asked for. One concern per commit.

  • A refused call asks for an answer. A call past tool_max_calls_per_turn got a role:"tool" reply saying only that it was not run, and the reply the assistant was then asked for without tools was often empty: it had planned more calls and had no answer ready, so the user got a blank turn. The refusal now reads "Not run: this turn reached its limit of N tool calls. Answer the user now with the results you have." (ToolExecutionMixin.after_assistant_turn), for every probe that uses the mixin's tool loop. architecture/probes.md and the mixin's docstring say so.
  • A message that asks for more never ends the conversation. The early-stop check asks the summary model whether a user message ends the conversation, and a model sometimes said yes to a question, or to a message giving the details the assistant had just asked for, so the conversation stopped with that message unanswered. A message holding a question mark (Latin, full-width or Arabic) now never ends it and costs no model call (_check_conversation_complete), and the prompt adds that a message answering the assistant's question, or giving it the details, a choice or a correction it asked for, does not end it even when it also thanks the assistant or says that is all it needs. Two test doubles whose scripted user asked a question while their summary model answered yes now script the request without a question mark, so they keep the flow they test.

No probe's prompt version changes: both changes are in the shared loop, as with earlier loop changes.

How it was verified

New tests fail on the code before each change and pass after it: the refusal's text for a call past the limit (test_probes_substrate.py); a question in English, French or Japanese not ending the conversation even when the summary model says yes, with no model call; and the prompt's new clause (test_engine_e2e.py).

On every commit's own tree: make check-all, the pre-commit hooks on the commit's files, pytest --collect-only (5273 and 5277 tests collected) and the engine and probe tests.

At the head:

$ make check-all
All checks passed!
471 files already formatted
$ make test
5277 passed in 63.48s (0:01:03)
$ make smoke
All checks passed.
$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
7 passed in 5.91s
7 passed in 1.86s

Live, against a custom provider config: the end-to-end health check with the tool_calling probe, in English; and the early-stop question, on main's prompt and on this one, for user messages that ended conversations too soon in earlier runs and two that should end them, asked of the config's summary model and of a second model:

$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=en_US
✓ simulate, eval and report all completed
# main's prompt
hi_Deva_IN, answered the assistant's question: expected no; summary_model said 'yes', second model said 'yes'
fr_FR, a new request: expected no; summary_model said 'no', second model said 'no'
en_US, a question: expected no; summary_model said 'no', second model said 'no'
en_US, a goodbye: expected yes; summary_model said 'yes', second model said 'yes'
ja_JP, a goodbye: expected yes; summary_model said 'yes', second model said 'yes'
# this prompt, two runs
hi_Deva_IN, answered the assistant's question: expected no; summary_model said 'no', second model said 'no'
hi_Deva_IN, answered the assistant's question: expected no; summary_model said 'no', second model said 'yes'
fr_FR, a new request: expected no; summary_model said 'no', second model said 'no' (both runs)
en_US, a question: question mark, not ended without a model call (both runs)
en_US, a goodbye: expected yes; summary_model said 'yes', second model said 'yes' (both runs)
ja_JP, a goodbye: expected yes; summary_model said 'yes', second model said 'yes' (both runs)

Results impact

Conversations that reach the tool-call limit should end the turn with an answer rather than a blank reply, so turns that a judge or scorer marked down for an empty reply change. Fewer conversations stop on a user message that asks for something: one holding a question mark always continues, and a message that answers the assistant continues more often, though a model can still misjudge one without a question mark. Conversations run a turn longer where they used to stop on such a message, and early_stop counts move with them. A user's goodbye that also asks a question now continues one more turn.

3mei added 2 commits November 20, 2026 00:02
A call past tool_max_calls_per_turn got a reply saying only that it was
not run, and the reply the assistant was then asked for without tools
was often empty: the assistant had planned more calls and had no answer
ready. The refusal now also asks the assistant to answer the user with
the results it has.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
… more

The early-stop check asked the summary model whether a user message
ends the conversation, and a model sometimes said yes to a question, or
to a message giving the details the assistant had asked for, so the
conversation stopped with that message unanswered. A message holding a
question mark now never ends a conversation and needs no model call,
and the check's prompt says a message answering the assistant, or giving
it the details, a choice or a correction it asked for, does not end it
either.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner November 20, 2026 05:02
@3mei
3mei merged commit 0cfa9c7 into main Nov 20, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant