You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Conformance: rules for what a user sees (failed tests, call sites, every test has rows) #454
The conformance suite checks how the stream is built (ordering, timing, which page a row replays, request counts), but not three things a user sees directly. Each of these broke in the Selenium adapter without any automated check noticing (fixed in #453):
A failed test can leave nothing red. A driver.wait() that timed out produced no row, so the Actions list and the Errors tab showed a test that simply stopped. failed-command-has-error-row covers failed commands, but a wait is not a command, and no rule asks whether a failed test explains its failure.
The call site can point at our own code. Under source maps (tsx, vitest) every Selenium row's callSource pointed at packages/core/src/stack.ts:63 instead of the test file. No rule checks where callSource points.
The first test can lose all its rows. Hooks registered on a 100 ms poll missed a first test that started sooner. Real browsers start slowly enough to hide it; nothing checks that every test in the scenario has rows.
What would you like DevTools to do?
Add three rules to the conformance suite, judged for every adapter:
failed-test-has-error-row: every test the runner reports as failed has at least one row in its window carrying an error.
call-site-names-the-scenario: every row with a callSource names the scenario file (and a line inside it), never a file under packages/.
every-test-has-rows: every test the scenario defines has at least one row.
Each needs scenario input to bite: a deliberately failing test (rule 1), and a run through the real test runner (rules 2 and 3), which the scenario issue covers. Rules an adapter cannot meet yet go in KNOWN_GAPS with a reason, and the capability matrix (packages/conformance/src/features.ts) can then mark the matching cells tested.
Which part of DevTools would this touch?
Conformance suite (packages/conformance).
Which frameworks should this apply to?
All four adapters: WebdriverIO, Selenium, Nightwatch, Python.
Anything you've tried or considered?
Unit tests caught none of the three: they feed hand-made stacks and call hooks directly, so they confirm each piece does what it was written to do, not what a user sees at the end of a run.
Activity