Repository navigation
Conformance rules for navigation timings and page-level assertion targets - #483
Merged
Merged
Conversation
…ce rules Two bugs fixed this week (#467, #476) were caught by review, not by the conformance suite, because no rule covered them. navigation-row-carries-timings: the fake browser now runs a script that reads getEntriesByType('navigation') against the page's own timings, title and url, as a WebDriver body, a BiDi function declaration or a BiDi expression, so it answers only what the script returns. A capture script that builds its payload and never returns it is caught the way a real browser exposes it. The rule judges every successful navigation row after its replaceCommand updates are applied, as the dashboard shows it. The script is matched by content, not by import path, so it survives moving the script out of core. page-level-assertion-names-no-element: a new scenario test reads the heading, reads the page title (which now reads the same), and asserts the title is not 'Secure Area', a value an element read earlier. Taking the target from either the colliding read or the expected value boxes the wrong element. The WebdriverIO service fails this rule on main, because it stamps node:assert rows with the last element it resolved and hides the user's own getTitle; that is declared as a scoped gap against #481 rather than fixed here. The service's rows carry a testUid the suite tree never reports, so the assertion that ends a test was placed by time and, sharing a millisecond with the next test's start, landed in the wrong test. Rows of a uid the tree does not report are now placed together by the first of them. The performance timings cells are now backed by the rule for all four adapters; the page-level rule is listed as internal, being one property of assertion capture.
Contributor
[Low impact] The PR appears safe to merge, with the declared service fixes supplied by PR #484. SummaryAdds conformance checks for navigation timings and page-title assertion targets.
Reviews (3) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile |
2 of 18 tasks
…the page's load time Main fixed Nightwatch reporting passing tests as skipped (#468), so the skipped gap this branch extended to the page-title test goes, with its debt entry and CLAUDE.md bullet; the service gap for #481 stays. The timings rule accepted any navigation entry, so an empty object passed. It now requires the load time the fake page reports, so a capture that drops its timing figures fails.
…timings-targets # Conflicts: # docs/ADAPTERS.md
4 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Closes #479.
Two bugs fixed this week, core's performance script never returning its payload (#467) and a title assertion boxing an element whose earlier read matched the expected value (#476, #475 for Python), were caught by review, not by the conformance suite. This adds a rule for each.
navigation-row-carries-timings: every successful navigation row carriesperformanceonce itsreplaceCommandupdates are applied (sharedreplaceCommand, the dashboard's own rule). The fake browser now recognises a script by itsgetEntriesByType('navigation')call, not by the exact source or an import path, and runs it in avmsandbox against the page's own timings, title and url. It does this for a WebDriver body, a BiDi function declaration (WebdriverIO'sexecuteover BiDi) and a BiDi expression, so it answers what the script returns and nothing if it returns nothing, as a browser does. A script that throws in the page now gets a WebDriverjavascript erroror a BiDi exception result instead of a dropped request.page-level-assertion-names-no-element: a new scenario test, Asserts on the page title, in all four scenarios in each runner's own idiom. It readsh2(Login Page), reads the page title (nowLogin Pagetoo, fromPAGE_TITLES), and asserts the title does not equalSecure Area, which test 2's condition wait read from the secure page'sh2. The rule says that assertion row carries noselector, and that the test shows at least one assertion row. A target taken from the colliding heading read (the no-element sentinel) or from the expected value (fix(core): name no target when an assert's subject belongs to no element #476) boxesh2.Capability matrix: the "Performance timings on navigation rows" cells are now
allTested([navigation-row-carries-timings])for all four adapters, which removes the by-hand reasons. The page-level rule is listed inINTERNAL_RULES: it is one property of assertion capture, and that feature is partial in two adapters. README regenerated.Type of change
Packages touched
shared(types and contracts)core(framework-agnostic capture/reporting)elements(published element/snapshot API,@wdio/elements)service(WebdriverIO adapter)nightwatch-devtools(Nightwatch adapter)selenium-devtools(Selenium adapter)selenium-devtools-py(Selenium Python adapter)backend(server)app(UI)script(page-injected runtime)trace(Trace mode)Only
packages/conformance, its four scenarios,docs/ADAPTERS.md,CLAUDE.mdand the generated README table.Capability matrix
packages/conformance/src/features.tsupdated, and tables regenerated withpnpm docs:capabilities, if this adds or changes user-facing behaviourThe docs-site copy (
CrossFrameworkSupport.md) will be refreshed withpnpm docs:capabilities --website <checkout>in a separate webdriverio PR after this merges.Notes for reviewers
The WebdriverIO service fails the page-level rule on main.
SessionCapturer.captureAssertCommandstamps anode:assertrow with#lastResolvedSelector, and the service never feeds core's value registry, so the title assertion carriesh2in all three BiDi modes. The user's ownbrowser.getTitle()also has no row, becausegetTitleis inINTERNAL_COMMANDS. That is a real adapter bug, not a suite assumption. It is filed as #481 and declared as two scoped gaps (the rule's one message, and the one agreement hunk), with a Known debt entry in CLAUDE.md, rather than fixed here, so this PR stays one concern.Row attribution by uid group. The service tags rows with a
testUidthe suite tree never reports, so its rows were placed by time alone. The new test ends in a synchronousnotEqualthat shared its millisecond with the next test's start, so it landed in the wrong test.scenarioTestsOfnow places all rows of an unreported uid by the first of them. Adapters whose uids match the tree are unaffected.Proof (each rule fails with its fix reverted, passes on main):
returnremoved from core'sCAPTURE_PERFORMANCE_SCRIPT(and Python's copy)navigation-row-carries-timingsonly (url(.../login)/get(.../login) never carries the page's navigation timings); the 3 agreement tests still passresolveAssertTargetFromArgspage-level-assertion-names-no-elementin all 3 modes (assert.notEqual on the page title names h2)read_value_locators.assertion_target_UNRESOLVEDguard inassertions.pydriver.titleis never re-read, and the scenario runs under pytestChecks:
pnpm build,pnpm test(3043 passed),pnpm lint(0 errors),pnpm typecheck,pnpm docs:capabilities --check, and the Python suite (792 passed. The 2 failures,test_an_abandoned_driver_is_not_retainedandtest_the_recorder_does_not_retain_its_driver, fail on main too).Side observation, not changed here: the service also attaches timings to clicks that navigate, and to the refused
/brokennavigation, where it reads the page the browser stayed on.Screenshots / recordings
Not applicable: no UI change.
🤖 Generated with Claude Code