Skip to content

Conformance: rules for what a user sees (failed tests, call sites, every test has rows) #454

Closed

Description

@vishnuv688

Part of #278. Follows #300.

What problem are you trying to solve?

The conformance suite checks how the stream is built (ordering, timing, which page a row replays, request counts), but not three things a user sees directly. Each of these broke in the Selenium adapter without any automated check noticing (fixed in #453):

  • A failed test can leave nothing red. A driver.wait() that timed out produced no row, so the Actions list and the Errors tab showed a test that simply stopped. failed-command-has-error-row covers failed commands, but a wait is not a command, and no rule asks whether a failed test explains its failure.
  • The call site can point at our own code. Under source maps (tsx, vitest) every Selenium row's callSource pointed at packages/core/src/stack.ts:63 instead of the test file. No rule checks where callSource points.
  • The first test can lose all its rows. Hooks registered on a 100 ms poll missed a first test that started sooner. Real browsers start slowly enough to hide it; nothing checks that every test in the scenario has rows.

What would you like DevTools to do?

Add three rules to the conformance suite, judged for every adapter:

  1. failed-test-has-error-row: every test the runner reports as failed has at least one row in its window carrying an error.
  2. call-site-names-the-scenario: every row with a callSource names the scenario file (and a line inside it), never a file under packages/.
  3. every-test-has-rows: every test the scenario defines has at least one row.

Each needs scenario input to bite: a deliberately failing test (rule 1), and a run through the real test runner (rules 2 and 3), which the scenario issue covers. Rules an adapter cannot meet yet go in KNOWN_GAPS with a reason, and the capability matrix (packages/conformance/src/features.ts) can then mark the matching cells tested.

Which part of DevTools would this touch?

Conformance suite (packages/conformance).

Which frameworks should this apply to?

All four adapters: WebdriverIO, Selenium, Nightwatch, Python.

Anything you've tried or considered?

Unit tests caught none of the three: they feed hand-made stacks and call hooks directly, so they confirm each piece does what it was written to do, not what a user sees at the end of a run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions