Skip to content

Conformance rules for navigation timings and page-level assertion targets - #483

Merged
vishnuv688 merged 3 commits into
mainfrom
feat/479-conformance-timings-targets
Oct 28, 2026
Merged

vishnuv688 merged 3 commits into
mainfrom
feat/479-conformance-timings-targets

Conversation

@vishnuv688

Copy link
Copy Markdown
Member

What & why

Closes #479.

Two bugs fixed this week, core's performance script never returning its payload (#467) and a title assertion boxing an element whose earlier read matched the expected value (#476, #475 for Python), were caught by review, not by the conformance suite. This adds a rule for each.

  • navigation-row-carries-timings: every successful navigation row carries performance once its replaceCommand updates are applied (shared replaceCommand, the dashboard's own rule). The fake browser now recognises a script by its getEntriesByType('navigation') call, not by the exact source or an import path, and runs it in a vm sandbox against the page's own timings, title and url. It does this for a WebDriver body, a BiDi function declaration (WebdriverIO's execute over BiDi) and a BiDi expression, so it answers what the script returns and nothing if it returns nothing, as a browser does. A script that throws in the page now gets a WebDriver javascript error or a BiDi exception result instead of a dropped request.
  • page-level-assertion-names-no-element: a new scenario test, Asserts on the page title, in all four scenarios in each runner's own idiom. It reads h2 (Login Page), reads the page title (now Login Page too, from PAGE_TITLES), and asserts the title does not equal Secure Area, which test 2's condition wait read from the secure page's h2. The rule says that assertion row carries no selector, and that the test shows at least one assertion row. A target taken from the colliding heading read (the no-element sentinel) or from the expected value (fix(core): name no target when an assert's subject belongs to no element #476) boxes h2.

Capability matrix: the "Performance timings on navigation rows" cells are now allTested([navigation-row-carries-timings]) for all four adapters, which removes the by-hand reasons. The page-level rule is listed in INTERNAL_RULES: it is one property of assertion capture, and that feature is partial in two adapters. README regenerated.

Type of change

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Polish (an improvement to an existing feature)
  • Breaking change (existing behavior changes for users)
  • Documentation
  • Internal (build, CI, dependencies, tooling)

Packages touched

  • shared (types and contracts)
  • core (framework-agnostic capture/reporting)
  • elements (published element/snapshot API, @wdio/elements)
  • service (WebdriverIO adapter)
  • nightwatch-devtools (Nightwatch adapter)
  • selenium-devtools (Selenium adapter)
  • selenium-devtools-py (Selenium Python adapter)
  • backend (server)
  • app (UI)
  • script (page-injected runtime)
  • trace (Trace mode)

Only packages/conformance, its four scenarios, docs/ADAPTERS.md, CLAUDE.md and the generated README table.

Capability matrix

  • packages/conformance/src/features.ts updated, and tables regenerated with pnpm docs:capabilities, if this adds or changes user-facing behaviour

The docs-site copy (CrossFrameworkSupport.md) will be refreshed with pnpm docs:capabilities --website <checkout> in a separate webdriverio PR after this merges.

Notes for reviewers

The WebdriverIO service fails the page-level rule on main. SessionCapturer.captureAssertCommand stamps a node:assert row with #lastResolvedSelector, and the service never feeds core's value registry, so the title assertion carries h2 in all three BiDi modes. The user's own browser.getTitle() also has no row, because getTitle is in INTERNAL_COMMANDS. That is a real adapter bug, not a suite assumption. It is filed as #481 and declared as two scoped gaps (the rule's one message, and the one agreement hunk), with a Known debt entry in CLAUDE.md, rather than fixed here, so this PR stays one concern.

Row attribution by uid group. The service tags rows with a testUid the suite tree never reports, so its rows were placed by time alone. The new test ends in a synchronous notEqual that shared its millisecond with the next test's start, so it landed in the wrong test. scenarioTestsOf now places all rows of an unreported uid by the first of them. Adapters whose uids match the tree are unaffected.

Proof (each rule fails with its fix reverted, passes on main):

Revert Result
none (this branch), 3 runs 15/15 conformance tests green each time
return removed from core's CAPTURE_PERFORMANCE_SCRIPT (and Python's copy) 12 of 12 adapter runs fail with navigation-row-carries-timings only (url(.../login) / get(.../login) never carries the page's navigation timings); the 3 agreement tests still pass
#476 in resolveAssertTargetFromArgs Selenium and Nightwatch fail page-level-assertion-names-no-element in all 3 modes (assert.notEqual on the page title names h2)
Python's equivalent stop in read_value_locators.assertion_target Python fails the same rule in all 3 modes
#475's _UNRESOLVED guard in assertions.py Python still passes: that guard covers plain scripts, where driver.title is never re-read, and the scenario runs under pytest

Checks: pnpm build, pnpm test (3043 passed), pnpm lint (0 errors), pnpm typecheck, pnpm docs:capabilities --check, and the Python suite (792 passed. The 2 failures, test_an_abandoned_driver_is_not_retained and test_the_recorder_does_not_retain_its_driver, fail on main too).

Side observation, not changed here: the service also attaches timings to clicks that navigate, and to the refused /broken navigation, where it reads the page the browser stayed on.

Screenshots / recordings

Not applicable: no UI change.

🤖 Generated with Claude Code

…ce rules

Two bugs fixed this week (#467, #476) were caught by review, not by the
conformance suite, because no rule covered them.

navigation-row-carries-timings: the fake browser now runs a script that
reads getEntriesByType('navigation') against the page's own timings, title
and url, as a WebDriver body, a BiDi function declaration or a BiDi
expression, so it answers only what the script returns. A capture script
that builds its payload and never returns it is caught the way a real
browser exposes it. The rule judges every successful navigation row after
its replaceCommand updates are applied, as the dashboard shows it. The
script is matched by content, not by import path, so it survives moving the
script out of core.

page-level-assertion-names-no-element: a new scenario test reads the
heading, reads the page title (which now reads the same), and asserts the
title is not 'Secure Area', a value an element read earlier. Taking the
target from either the colliding read or the expected value boxes the wrong
element. The WebdriverIO service fails this rule on main, because it stamps
node:assert rows with the last element it resolved and hides the user's own
getTitle; that is declared as a scoped gap against #481 rather than fixed
here.

The service's rows carry a testUid the suite tree never reports, so the
assertion that ends a test was placed by time and, sharing a millisecond
with the next test's start, landed in the wrong test. Rows of a uid the
tree does not report are now placed together by the first of them.

The performance timings cells are now backed by the rule for all four
adapters; the page-level rule is listed as internal, being one property of
assertion capture.
@vishnuv688 vishnuv688 self-assigned this Oct 28, 2026
@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge, with the declared service fixes supplied by PR #484.

Summary

Adds conformance checks for navigation timings and page-title assertion targets.

  • The conformance suite checks that navigation rows keep the page's load timings.
  • The conformance suite checks that page-title assertions name no element.

Reviews (3) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

Comment thread packages/conformance/src/checker.ts Resolved
…the page's load time

Main fixed Nightwatch reporting passing tests as skipped (#468), so the
skipped gap this branch extended to the page-title test goes, with its
debt entry and CLAUDE.md bullet; the service gap for #481 stays.

The timings rule accepted any navigation entry, so an empty object
passed. It now requires the load time the fake page reports, so a
capture that drops its timing figures fails.
…timings-targets

# Conflicts:
#	docs/ADAPTERS.md
@vishnuv688
vishnuv688 merged commit 04beaa0 into main Oct 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conformance rules for navigation timings and page-level assertion targets

1 participant