Skip to content

Run a round's tool calls at once in every provider #375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattt merged 4 commits into from
Nov 6, 2026
Merged

Run a round's tool calls at once in every provider #375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a3c09eb
Run a round's tool calls at once in every provider
gtokman Nov 6, 2026
ed5aadb
Merge remote-tracking branch 'origin/main' into mattt/gary-concurrent…
mattt Nov 6, 2026
c62a40f
Keep one entry for each MLX and llama.cpp tool round
mattt Nov 6, 2026
812472d
Correct the tool callback and history docs for MLX and llama.cpp
mattt Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 28 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,14 @@ let response = try await session.respond {
print(response.content)
```

When a model calls several tools in one round,
the calls run at once, as they do in Foundation Models,
and the round takes as long as its slowest call.
A tool can be called concurrently with itself or with other tools,
so one that isn't safe to run alongside others should serialize its own work.
Results go back to the model in the order it made the calls.
If a call throws, the others are cancelled and the response fails with that error.

To observe or control tool execution, assign a delegate on the session:

```swift
Expand Down Expand Up @@ -504,6 +512,12 @@ actor ToolExecutionObserver: ToolExecutionDelegate {
session.toolExecutionDelegate = ToolExecutionObserver()
```

The delegate decides each call in the order the model made them, before any call runs.
Once the round's calls are done,
it hears about each one in that order, one at a time.
If a call throws, the others are cancelled first,
and the delegate hears only about the error the response fails with.

When a delegate returns `.stop`,
the response ends with the tool calls that stopped it.
`respond` returns an empty value,
Expand Down Expand Up @@ -773,10 +787,7 @@ Use `session.properties` in those places.

The five HTTP providers (OpenAI, Open Responses, Anthropic, Gemini, and Ollama)
evaluate the profile and its history transforms again
before each request that continues a response after tool calls,
and run `onToolCall` and `onToolOutput`.
With these providers, a Tool's `history` property includes
the tool calls and outputs of the current response, including the Tool's own call.
before each request that continues a response after tool calls.

Create a profile's model outside `body`, as `EditingProfile` does.
Each model that these providers create has its own identity, which copies share,
Expand All @@ -786,11 +797,20 @@ and each continuation hands the response off to the new one.
With the OpenAI Responses API and Open Responses,
the next request then leaves out the reasoning item and the function call's `id`.

MLX, llama.cpp, Core ML, Private Cloud Compute, `FoundationLanguageModel`,
and the system model don't run `onToolCall` or `onToolOutput` yet.
MLX and llama.cpp evaluate the profile once for each response,
so their tool rounds use the same instructions, tools, and options,
and a Tool's `history` property doesn't include the response's tool rounds.
so their tool rounds use the same instructions, tools, options,
and transcript projection.

The five HTTP providers, MLX, and llama.cpp run `onToolCall` and `onToolOutput`.
With these providers, a Tool's `history` property includes
the calls and outputs of the response's earlier tool rounds
and every call in the current round, including the Tool's own call.
It doesn't include the outputs of the other calls in the current round,
even the ones that have finished, because the calls in a round run at once.
A Tool in a program that uses Foundation Models 27 directly sees the same entries.

Core ML, Private Cloud Compute, `FoundationLanguageModel`,
and the system model don't run `onToolCall` or `onToolOutput` yet.

The system model, Private Cloud Compute, and `FoundationLanguageModel`
run tools in a native Foundation Models session.
Expand Down
18 changes: 9 additions & 9 deletions Sources/AnyLanguageModel/DynamicProfile.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ extension LanguageModelSession {
/// OpenAI, Open Responses, Anthropic, Gemini, and Ollama also evaluate it again
/// before each request that continues a response after a tool call.
/// MLX and llama.cpp evaluate it once for each response,
/// so their tool rounds use the same instructions, tools, and options.
/// so their tool rounds use the same instructions, tools, options, and transcript projection.
/// The system model, Private Cloud Compute, and `FoundationLanguageModel` evaluate it again
/// before the continuation after a tool round,
/// but a continuation that selects another model stays on the native session's model,
Expand Down Expand Up @@ -634,8 +634,8 @@ extension LanguageModelSession.DynamicProfile {
/// - Parameter action: The action to run, which receives the tool call.
/// - Returns: A profile with the action.
///
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, and Ollama run this action
/// when they call a tool. MLX, llama.cpp, Core ML, Private Cloud Compute,
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, Ollama, MLX, and llama.cpp
/// run this action when they call a tool. Core ML, Private Cloud Compute,
/// `FoundationLanguageModel`, and the system model don't run it yet.
public func onToolCall(
@_inheritActorContext perform action:
Expand All @@ -654,8 +654,8 @@ extension LanguageModelSession.DynamicProfile {
/// - Parameter action: The action to run.
/// - Returns: A profile with the action.
///
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, and Ollama run this action
/// when they call a tool. MLX, llama.cpp, Core ML, Private Cloud Compute,
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, Ollama, MLX, and llama.cpp
/// run this action when they call a tool. Core ML, Private Cloud Compute,
/// `FoundationLanguageModel`, and the system model don't run it yet.
public func onToolCall(
@_inheritActorContext perform action: nonisolated(nonsending) sending @escaping () async throws -> Void
Expand All @@ -670,8 +670,8 @@ extension LanguageModelSession.DynamicProfile {
/// - Parameter action: The action to run, which receives the tool call and its output.
/// - Returns: A profile with the action.
///
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, and Ollama run this action
/// when they call a tool. MLX, llama.cpp, Core ML, Private Cloud Compute,
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, Ollama, MLX, and llama.cpp
/// run this action when they call a tool. Core ML, Private Cloud Compute,
/// `FoundationLanguageModel`, and the system model don't run it yet.
public func onToolOutput(
@_inheritActorContext perform action:
Expand All @@ -691,8 +691,8 @@ extension LanguageModelSession.DynamicProfile {
/// - Parameter action: The action to run.
/// - Returns: A profile with the action.
///
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, and Ollama run this action
/// when they call a tool. MLX, llama.cpp, Core ML, Private Cloud Compute,
/// - Important: OpenAI, Open Responses, Anthropic, Gemini, Ollama, MLX, and llama.cpp
/// run this action when they call a tool. Core ML, Private Cloud Compute,
/// `FoundationLanguageModel`, and the system model don't run it yet.
public func onToolOutput(
@_inheritActorContext perform action: nonisolated(nonsending) sending @escaping () async throws -> Void
Expand Down
143 changes: 39 additions & 104 deletions Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -710,92 +710,33 @@ import Foundation
}
}

/// Resolves a round of tool calls.
///
/// The returned entry is the one that callbacks and tools saw,
/// so the caller records it rather than a new entry with a different ID.
private func resolveToolCalls(
_ parsedCalls: [LlamaParsedToolCall],
tools: [any Tool],
requestContext: LanguageModelSession.RequestContext,
offeredTools: [any Tool],
currentRoundEntries: [Transcript.Entry],
session: LanguageModelSession
) async throws -> ToolResolutionOutcome {
if parsedCalls.isEmpty { return .invocations([]) }

var toolsByName: [String: any Tool] = [:]
for tool in tools where toolsByName[tool.name] == nil {
toolsByName[tool.name] = tool
}

) async throws -> (calls: Transcript.ToolCalls, outcome: ToolResolutionOutcome) {
let transcriptCalls = try makeTranscriptToolCalls(from: parsedCalls)

if let delegate = session.toolExecutionDelegate {
await delegate.didGenerateToolCalls(transcriptCalls, in: session)
}

var decisions: [ToolExecutionDecision] = []
decisions.reserveCapacity(transcriptCalls.count)

if let delegate = session.toolExecutionDelegate {
for call in transcriptCalls {
let decision = await delegate.toolCallDecision(for: call, in: session)
if case .stop = decision {
return .stop(calls: transcriptCalls)
}
decisions.append(decision)
}
} else {
decisions = Array(repeating: .execute, count: transcriptCalls.count)
}

var results: [ToolInvocationResult] = []
results.reserveCapacity(transcriptCalls.count)

for (index, call) in transcriptCalls.enumerated() {
switch decisions[index] {
case .stop:
return .stop(calls: transcriptCalls)
case .provideOutput(let segments):
let output = Transcript.ToolOutput(
id: call.id,
toolName: call.toolName,
segments: segments
)
if let delegate = session.toolExecutionDelegate {
await delegate.didExecuteToolCall(call, output: output, in: session)
}
results.append(ToolInvocationResult(call: call, output: output))
case .execute:
guard let tool = toolsByName[call.toolName] else {
let message = Transcript.Segment.text(.init(content: "Tool not found: \(call.toolName)"))
let output = Transcript.ToolOutput(
id: call.id,
toolName: call.toolName,
segments: [message]
)
if let delegate = session.toolExecutionDelegate {
await delegate.didExecuteToolCall(call, output: output, in: session)
}
results.append(ToolInvocationResult(call: call, output: output))
continue
}

do {
let segments = try await tool.makeOutputSegments(from: call.arguments)
let output = Transcript.ToolOutput(
id: call.id,
toolName: tool.name,
segments: segments
)
if let delegate = session.toolExecutionDelegate {
await delegate.didExecuteToolCall(call, output: output, in: session)
}
results.append(ToolInvocationResult(call: call, output: output))
} catch {
if let delegate = session.toolExecutionDelegate {
await delegate.didFailToolCall(call, error: error, in: session)
}
throw LanguageModelSession.ToolCallError(tool: tool, underlyingError: error)
}
}
let callsEntry = Transcript.ToolCalls(transcriptCalls)
let entries = currentRoundEntries + [.toolCalls(callsEntry)]
switch try await session.resolveProfileToolCalls(
transcriptCalls,
requestContext: requestContext,
currentRoundEntries: entries,
offeredTools: offeredTools
) {
case .stop(let calls):
return (callsEntry, .stop(calls: calls))
case .invocations(let invocations):
return (
callsEntry, .invocations(invocations.map { ToolInvocationResult(call: $0.call, output: $0.output) })
)
}

return .invocations(results)
}

public func respond<Content>(
Expand Down Expand Up @@ -945,16 +886,16 @@ import Foundation

try session.recordToolRound(toolRounds, completed: allEntries)

let resolution = try await resolveToolCalls(
let (callsEntry, resolution) = try await resolveToolCalls(
parsedCalls,
tools: requestContext.toolsForExecution(in: session, offeredTools: tools),
requestContext: requestContext,
offeredTools: tools,
currentRoundEntries: allEntries,
session: session
)
switch resolution {
case .stop(let calls):
if !calls.isEmpty {
allEntries.append(.toolCalls(Transcript.ToolCalls(calls)))
}
case .stop:
allEntries.append(.toolCalls(callsEntry))
return try stoppedResponse(
generating: type,
provider: "Llama",
Expand All @@ -965,11 +906,8 @@ import Foundation
guard !invocations.isEmpty else {
break generationLoop
}
let callsEntry = Transcript.Entry.toolCalls(
Transcript.ToolCalls(invocations.map(\.call))
)
allEntries.append(callsEntry)
pendingEntries.append(callsEntry)
allEntries.append(.toolCalls(callsEntry))
pendingEntries.append(.toolCalls(callsEntry))
for invocation in invocations {
let outputEntry = Transcript.Entry.toolOutput(invocation.output)
allEntries.append(outputEntry)
Expand Down Expand Up @@ -1198,29 +1136,26 @@ import Foundation

try session.recordToolRound(toolRounds, completed: accumulatedEntries)

let resolution = try await self.resolveToolCalls(
let (callsEntry, resolution) = try await self.resolveToolCalls(
parsedCalls,
tools: requestContext.toolsForExecution(in: session, offeredTools: tools),
requestContext: requestContext,
offeredTools: tools,
currentRoundEntries: accumulatedEntries,
session: session
)
switch resolution {
case .stop(let calls):
case .stop:
emittedBase += roundVisible
if !calls.isEmpty {
accumulatedEntries.append(.toolCalls(Transcript.ToolCalls(calls)))
yieldSnapshot(emittedBase)
}
accumulatedEntries.append(.toolCalls(callsEntry))
yieldSnapshot(emittedBase)
break generationLoop
case .invocations(let invocations):
guard !invocations.isEmpty else {
emittedBase += roundVisible
break generationLoop
}
let callsEntry = Transcript.Entry.toolCalls(
Transcript.ToolCalls(invocations.map(\.call))
)
accumulatedEntries.append(callsEntry)
pendingEntries.append(callsEntry)
accumulatedEntries.append(.toolCalls(callsEntry))
pendingEntries.append(.toolCalls(callsEntry))
for invocation in invocations {
let outputEntry = Transcript.Entry.toolOutput(invocation.output)
accumulatedEntries.append(outputEntry)
Expand Down
Loading
Loading