[vishnuv688]

What & why

Closes #455

Stacked on #453 (base fix/selenium-rows-after-wait): the Selenium scenario relies on #453's mocha root hooks and failed-wait row. Retarget to main once #453 merges.

The conformance scenario was one short login in a single test, and Selenium and Python ran it as a plain script. Element waits, failing tests, test boundaries and hook registration under a real runner were never exercised, which is where the bugs fixed in #453 lived. Each adapter now runs five equivalent tests through its own runner:

1. logs in and reads the greeting (passes): fills and submits the form, waits for a message the fake page renders 150 ms late (so the wait polls), reads it and asserts on it.
2. waits on what the page shows (passes): a wait whose condition reads two elements, and a helper wait repeating the test's own read, then logs out.
3. fails on the wrong heading (fails): an assertion on the wrong text.
4. times out waiting for a missing element (fails): a short wait for an element that never renders.
5. survives a refused navigation (passes).

Adapter	Runner
WebdriverIO	WDIO runner
Nightwatch	Nightwatch CLI
Selenium (Node.js)	mocha with --require @wdio/selenium-devtools
Python	pytest with --devtools

Checks that come with it

• Exit codes: each runner must exit with its own code for two failed tests, and the new tests-end-as-designed rule checks every test's reported state, so a crash, a usage error or a should-pass test failing breaks the suite.
• Per-test step alignment: the cross-adapter comparison aligns steps within each test by their longest common sequence and reports a step shown at a different place as one move, so one excused difference no longer hides every later step.
• One definition of a failed row: shared/src/command-failure.ts (moved from the trace exporter, behaviour unchanged) is used by both the trace exporter and the comparison.
• Scoped gaps: a gap names exactly the difference it excuses. A declared lack goes stale when the adapter shows any step in its place, or when nothing it is compared with shows the step either. A gap matched against an adapter that did not run is not judged. Every gap points at a CLAUDE.md Known debt entry and an issue, and known-debt.test.ts enforces it.
• replace-targets-a-sent-row locates a row the way the dashboard does (shared replaceCommandIndex).
• The HTML report reads a run that exits as designed as passing.

Type of change

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Polish (an improvement to an existing feature)
• Breaking change (existing behavior changes for users)
• Documentation
• Internal (build, CI, dependencies, tooling)

Packages touched

• shared (types and contracts)
• core (framework-agnostic capture/reporting)
• elements (published element/snapshot API — @wdio/elements)
• service (WebdriverIO adapter)
• nightwatch-devtools (Nightwatch adapter)
• selenium-devtools (Selenium adapter)
• selenium-devtools-py (Selenium Python adapter)
• backend (server)
• app (UI)
• script (page-injected runtime)
• trace (Trace mode)

The shared/trace change only moves the failed-row predicate (commandFailure) into shared; the trace exporter's behaviour is unchanged and both packages are private and inlined, so no changeset. The rest is packages/conformance (private), CI (selenium-devtools-py[test] for pytest) and docs.

Notes for reviewers

Adapter bugs the scenarios exposed (recorded as narrowly scoped gaps with Known debt entries, not fixed here):

• Nightwatch: an awaited waitForElementPresent returns before the wait runs when the plugin is loaded #457 Nightwatch: an awaited waitForElementPresent returns before the wait runs when the plugin is loaded (15 of 15 runs).
• Nightwatch: a late screenshot replace can overwrite an assertion row stamped in the same millisecond #458 Nightwatch: a late screenshot replace overwrites an assertion row stamped in the same millisecond.
• Nightwatch: a passing test is reported as skipped #459 Nightwatch: passing tests that make no assertion are reported as skipped.
• Python: a WebDriverWait leaves no row, so a timed-out wait shows nothing red #461 Python: a WebDriverWait leaves no row. Python: assertion rows carry no element target #462 Python: assertion rows carry no element target.
• Waits render differently across adapters (no row, a condition row, or the reads) #464 Waits render differently across adapters.

Proof that the suite guards #453: with #453's latest commit reverse-applied, the Selenium scenario reports the merged condition-wait reads as an unexplained divergence. The full run with all of #453 reverted is posted below once #454 is stacked.

Two gaps are timing-dependent (Nightwatch's "logs in" reported failed under #457, and the #458 overwrite), so they are never reported stale and must be removed by hand when those issues are fixed; docs say so.

Checks: pnpm test 2884 passed, the conformance file passed 3 of 3 runs, pnpm typecheck clean, pnpm lint 0 errors.

Screenshots / recordings

n/a (no UI change)

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge with the verified fixes in dependent PR #466.

Summary

Runs each adapter’s five-test scenario through its own test runner.

• Each adapter runs the same five tests through its own runner.
• The harness checks that each scenario test ends in its planned state.
• Adapter comparisons follow each test and isolate the differences they excuse.
• The trace exporter and comparison use one shared rule for failed command rows.
• Replace events follow a row’s latest timestamp when checking its target.

The previous moved-step finding is fixed in the current code. This assessment depends on PR #466 for the missing Nightwatch wait-order gaps and additional checks that failed tests contain failed action rows.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Harness[Conformance harness] --> Runner[Adapter test runner]
  Runner --> Fake[Fake WebDriver]
  Runner --> Frames[Recorded frames]
  Frames --> States[Check test states]
  Frames --> Normalize[Group steps by test]
  Normalize --> Compare[Compare aligned and moved steps]
  States --> Gaps[Apply scoped known gaps]
  Compare --> Gaps
  Gaps --> Report[Conformance report]

Loading

_{Reviews (4) · Last reviewed commit: "test(conformance): drop the Nightwatch w..." · Reviewed by Greptile}