Skip to content

Conformance: realistic scenarios run through each adapter's real test runner #455

Closed

Description

@vishnuv688

Part of #278. Follows #300.

What problem are you trying to solve?

The conformance scenario is one short login flow, about five commands, in a single test, and for Selenium and Python it runs as a plain script rather than through a test runner. Real suites look different, and the bugs fixed in #453 lived exactly in that difference:

  • No element waits. No scenario calls driver.wait(until.elementLocated(...)) or an equivalent, so the wait-poll folding and a timed-out wait were never exercised.
  • No failing test and no second test. Failure reporting and per-test boundaries are only exercised by unit tests.
  • No test runner for Selenium or Python. A plain script skips hook registration entirely, which is where the first-test race was: under mocha --require the plugin loads before mocha publishes its globals.

What would you like DevTools to do?

Extend every adapter's scenario, keeping them equivalent so the cross-adapter agreement checks still apply:

  • Run each through its real runner: WDIO's runner (already), the Nightwatch CLI, mocha with --require @wdio/selenium-devtools for Selenium, and pytest for Python.
  • Several tests, including one that fails on an assertion and one whose element wait times out.
  • An element wait that succeeds, element reads (getText) and assertions on them.
  • A fast fake browser on purpose, so timing races that a real browser hides show up.

The fake WebDriver (packages/conformance/src/fake-webdriver.ts) needs whatever endpoints the new steps use (for example findElements returning nothing until a delay, so a wait polls).

Which part of DevTools would this touch?

Conformance suite (packages/conformance/scenarios, the fake WebDriver).

Which frameworks should this apply to?

All four adapters.

Anything you've tried or considered?

Adding the steps to one adapter's scenario only makes the "adapters agree" checks fail on the difference, so all four have to grow together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions