Repository navigation
Tool calling in the local server (1.1.0) - #4
Merged
Merged
Conversation
…ive formats per model (1.1.0) - Tools.swift: order-preserving JSON (Python json.dumps formatting), ToolDefinition/ToolCall, ToolCallParser (Hermes, Llama 3.1+, Mistral v3), ToolStreamGate for streaming. - ChatTemplate: tool rendering matching the official Qwen2.5/Qwen3/Llama 3.2 templates and mistral-common v3 byte for byte (golden tests); tool/assistant-call history; forced-call prefill. - Engine: tools and prefill through context trimming, tool-call marker tokens revealed only with tools, constrained function-name decoding for tool_choice "required". - Server: request validation, OpenAI response and streaming shapes for both APIs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🚀 Deploying Preview to Cloudflare 🚀Preview Deployments by commit
|
- A forced call cut off by max_tokens reports finish_reason "length" instead of a 500. - The function-name constraint runs through the comma after the name, so a forced call cannot close before its arguments; masking saves the allowed logits instead of hashing every id. - Whole-body key-ordered parse allows 256 levels; the 64-level cap applies to tools only. - One table of revealed tool-call markers; markers are stripped from unparsed output. - Removed unused JSONValue.from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Tool calling for the OpenAI-compatible server, which previously rejected any request with tools.
tools,tool_choice(auto/none/required/ named),parallel_tool_calls, assistanttool_calls,toolmessages. Responses usecontent: nullwhen only calls are made andfinish_reason: "tool_calls". Streaming sends id, type and name first, then the arguments.function_call/function_call_outputinput items,function_calloutput items withoutput_item.added,function_call_arguments.delta/.doneandoutput_item.doneevents.tools,tool_choiceandparallel_tool_callsare echoed in the response object.transformers.apply_chat_templateon the official Qwen2.5 / Qwen3 / Llama 3.2 templates and withmistral-common1.13.tool_choice: required: the call opener is prefilled and the function name is constrained to the defined tools during decoding. Named choices prefill the name.tool_usetemplate is read from the named template list.toolresults must answer an earlier call, nesting limit of 64.functions/function_callget a 400.Testing
swift test: 111 tests pass, 28 of them new (golden templates, parser, streaming gate, JSON, name constraint, request validation).openaiPython SDK 3.42.0, 17 checks per model: Qwen3 0.6B, Qwen3 8B, Qwen2.5 0.5B and Mistral 7B pass 17/17. Llama 3.2 1B passes 15/17: after a tool result it calls another tool, andmlx-lmwith the official template does the same, so this is the model's limit.🤖 Generated with Claude Code