Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Benchmark variants now run interleaved per scenario. Adding the page screenshot to the no-gate controller lowered verified passes from 109/150 to 79/150 on Jev 1.14. The stepwise controller (settling, compact history, per-field values, finish vs stop, no-op skipping, ordered aim progress, Escape) reached 150/150 with screenshots and 149/150 without. Three verifiers now check outcomes rather than the final page, and the fault matrix shows no pass with a broken outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EndformPassed on
|
Clairic, Cloudflare's extraction model, answers text fields as spans of the input, so the loop needs no candidate actions: it asks for the next element ref, the value to type, a direct URL, and a quote proving completion, and code acts by the element's role. In one interleaved batch the naive Jev option menu verified 110/150, naive Clairic 42/150, Clairic with a screenshot 30/150, and Clairic with Clef judging completion 53/150. Clairic mostly quoted the aim itself as completion evidence, and without that it did not know when to stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clairic with Clef also passed activity-after-account-update 10/10, so the 2/10 ceiling for multi-step scenarios applies to the naive loop only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
bun run jev:test