Repository navigation
Use a cloud based model - #1096
Open
zetter-rpf wants to merge 6 commits into
Open
zetter-rpf wants to merge 6 commits into
zetter-rpf wants to merge 6 commits into
Conversation
The responses API is more recent has support for more features such as tool use, however it isn't supported as widely by cloud providers. Specifically, the qwen models we're considering that is hosted by Cloudflare only supports the chat API. I think the chat API is sufficient for our use case. This also makes the mocking easier in the tests with a helper function. Responses vs chat API: https://developers.openai.com/api/docs/guides/migrate-to-responses Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This qwen model is a suggestion as it's one of the closest to the ones we have previously tested and has a reasonable cost. We will run the evaluator against it to confirm it's suitable. Note I took out the reasoning config as it changes the structure of the response and causes errors (I'm not sure why). I'll add this back in a future commit
I thought the other method was getting quite complex
When testing we found that reasoning increased cost and time while providing little benefit for the kinds of data we expect. Initially I tried disabling this using the 'reasoning' parameter but this changes the structure of the response. I've not been able to work out why, but assume it is the way this particular model is hosted or configured. Instead I've switched to an alternative way to disable reasoning using the /no_think command.
Having a low temperature makes the models relatively more predictable which is preferred for our needs. I tested this by asking the model to name an animal. I still saw variance in responses with a low temperature, but I saw less than with a higher temperature so I assume it's working.
In english, it's common to estimate each token as approximately 4 characters long so should be set to about 1/4 of the max result length. I've added a generous buffer on top of this.
Test coverage94.18% line coverage reported by SimpleCov. |
zetter-rpf
marked this pull request as ready for review
November 6, 2026 14:53
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — This switches friendly-error inference from a local model to Cloudflare, sending learner code and error traces externally when enabled. Before production activation, confirm the provider and gateway’s data-handling, logging and retention controls are appropriate for learner submissions. Needs human approval because:
You can add or adjust custom eligibility rules. Learn more. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status
What's changed?
I've adapted what we've already done to use a Cloudflare hosted model instead of a local one
See commits for more
Steps to perform after deploying to production
BYPASS_FEM_INFERENCE