Skip to content

Let a tool delegate add messages between tool rounds - #376

Merged
mattt merged 3 commits into
huggingface:mainfrom
gtokman:steering
Nov 6, 2026
Merged

mattt merged 3 commits into
huggingface:mainfrom
gtokman:steering

Conversation

@gtokman

@gtokman gtokman commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Resolves #360.

ToolExecutionDelegate.pendingMessages(in:) is asked after each round of tool calls, before the model's next request. Someone can steer a long response while it runs. The policy around it (an inbox, an acknowledgment, what interrupts the final text) stays in the app. The default adds nothing.

Following your notes on the issue:

  • It returns [Prompt], so a message can carry images.
  • Every provider whose tool loop continues after a round asks, through shared code. That's every provider with a tool loop: OpenAI (Chat Completions and Responses), Open Responses, Gemini, Anthropic, Ollama, MLX and llama.cpp, streaming or not. SystemLanguageModel doesn't ask, since Foundation Models runs its own tool loop, and the method's documentation says so.
  • Steering doesn't reset the tool-round limit.
  • Each message's transcript entry is built once and sent in the same form. The entry is built by pendingPrompts (Shared/PendingMessages.swift). Each provider sends it with the same conversion it replays a transcript's prompts with, pulled into one function where that was inline, as Replay OpenAI tool calls as plain, stable JSON #325 does for tool calls. A request made now and the replay of it on a later turn match, so a prompt cache still matches.
  • Failure is documented. Once handed over, the messages are the response's. If it fails, they follow the session's transcriptErrorHandlingPolicy like its other entries, and they aren't handed back.
  • Messages still pending when a response ends without another round stay with the app, to send as the next prompt.

Tests

  • OpenAI, streaming and not: the message follows the tool output, is in the transcript as a prompt, and is sent in flight just as the next turn replays it (compared as decoded JSON).
  • OpenAI image: a message can carry an image.
  • OpenAI round limit: steering doesn't reset it, and the error's transcript entries include the steering prompt.
  • Anthropic and Ollama: streaming and not.
  • Chat Completions, Responses, Open Responses and Gemini: one parameterized test.
  • Not covered offline: MLX and llama.cpp build with their traits, but their steering paths need a local model to test.

The full suite has the same failures as main here: the 13 tests that need live services.

Rebased on main. This touches some of the files #375 does, and the two branches merge cleanly.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B4WBsc1Jp8fGufe89MhQFB

gtokman added a commit to gtokman/AnyLanguageModel that referenced this pull request Oct 29, 2026
`ToolExecutionDelegate.pendingMessages(in:)` returns `[Prompt]` rather
than `[String]`, as huggingface#376 has it. Each
message's prompt entry is built once, with the prompt's own transcript
segments, and sent with the same conversion as the transcript's
prompts, so a photo sent while a response runs goes with its message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4WBsc1Jp8fGufe89MhQFB

@mattt mattt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @gtokman. This follows what we discussed on #360, and returning [Prompt] works out nicely.

Two non-blocking notes:

  • The description says the message is byte-for-byte the same in flight and on replay, but the test compares decoded dictionaries. Either is fine with me; the test and the description should just agree.
  • The round-limit test could also check that the error's transcript entries include the steering prompt.

`ToolExecutionDelegate.pendingMessages(in:)` is asked after each round of
tool calls, before the model's next request, and returns `[Prompt]`, so a
message can carry images. Someone can steer a long response while it
runs. The default adds nothing.

Each message's prompt entry is built once, by shared code
(`pendingPrompts`), and every provider whose tool loop continues after a
round sends it with the same conversion it replays a transcript's
prompts with, as huggingface#325 does for tool calls: the request is the same now
and when a later response replays it, so a prompt cache still matches.
That's every one with a tool loop: OpenAI (Chat Completions and
Responses), Open Responses, Gemini, Anthropic, Ollama, MLX and
llama.cpp, streaming or not. `SystemLanguageModel` documents
that it doesn't ask, as Foundation Models runs its own tool loop.

Steering doesn't reset the tool-round limit. Messages the delegate has
handed over are the response's: if it fails, they follow the session's
`transcriptErrorHandlingPolicy` like its other entries, which the
method's documentation says. Messages still waiting when a response ends
without another round stay with the app, which can send them next.

Resolves huggingface#360.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4WBsc1Jp8fGufe89MhQFB
@gtokman

gtokman commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, @mattt. Both done, and the branch is rebased on main:

  • Description and test: the description now says the test compares the message as decoded JSON, which is what it checks.
  • Round-limit test: it now checks that the error's transcript entries end with the steering prompt, after the first round's output.

With the rebase, Anthropic's and Ollama's nonstreaming responses now run a tool loop too, so they ask for pending messages as well. Their tests cover both paths.

mattt added 2 commits November 6, 2026 09:08
Replay treated each prompt as the start of a turn, so a message from
pendingMessages(in:) ended the turn early. The rounds before the message lost
their text on a later turn, which breaks "each piece of text is sent once and
in order" from huggingface#401. Mark these prompts, and keep the mark in saved transcripts,
because a prompt's position can't separate them from a later turn's prompt.
@mattt

mattt commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks, @gtokman. #401 went in for 1.2.0 after your rebase, so I merged main into this branch to resolve the conflicts. It needed one more small change: a message from pendingMessages(in:) now stays in its response's turn on replay, through an internal continuesResponse property on Transcript.Prompt (0706398). Let me know what you think!

@mattt
mattt merged commit ab6aaf8 into huggingface:main Nov 6, 2026
14 checks passed
@mattt

mattt commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

This is now available in 1.3.0. Thanks again, @gtokman!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add messages to a response while it runs (steering between tool rounds)

2 participants