feat(conformance): judge failed tests, call sites and per-test rows

README.md

@@ -50,7 +50,7 @@ Want a portable artifact instead of a live UI (CI / agent diffing)? Switch to **
 | One row per command in the Actions list | ✅ tested | ✅ tested | ✅ tested | ✅ tested |
 | Screenshot on every command row | ✅ | ✅ | ✅ | ✅ |
 | Browser console logs | ✅ | ✅ | ✅ | ✅ |
-| Source tab opens the test line that ran the command | ✅ | ✅ | ✅ | ✅ |
+| Source tab opens the test line that ran the command | ✅ tested | ✅ tested | ✅ tested | ✅ tested |
 | Metadata tab (session, capabilities, device) | ✅ | ✅ | ✅ | ✅ |
 | Performance timings on navigation rows | ✅ | ✅ | ✅ | ✅ |
 | Failed command in the Errors tab, with the browser's message | ✅ tested | ✅ tested | ✅ tested | ✅ tested |

docs/ADAPTERS.md

@@ -238,7 +238,7 @@ Use your framework's own idiom for each step (its element wait, its text read, i
 
 1. Add your id to `AdapterId` in `src/types.ts`.
 2. Add `scenarios/<lang>/` holding a spec for the tests above, plus a config where your runner needs one (`scenarios/nightwatch/` and `scenarios/wdio/` are the models: the config reads the fake's origin from `CONFORMANCE_WEBDRIVER` and points the runner at it, with `start_process: false` for Nightwatch and `hostname`/`port` for WebdriverIO). Read the three variables by name (every spec repeats them, since none imports `SCENARIO_ENV`).
-3. Add one entry to `ADAPTERS` in `src/adapters.ts`: `exitCode` (above), `ready()` (your toolchain and build are present, including any plugin registration your runner needs, such as pytest's entry point) and `prepare()` (the command, args and env that launch your scenario, plus an optional `cleanup`, and a `cwd` for a CLI that resolves its runner and plugins from the working directory, as WebdriverIO does). `tests/conformance.test.ts` then runs your adapter in all three BiDi modes, holds it to every rule, and compares it with every other adapter. Expect the first run to be red: classify each line as a suite assumption to fix, a difference to declare with a reason, or an adapter bug to fix, and merge only when it is green. `ready()` may only skip a local run; in CI a missing prerequisite fails.
+3. Add one entry to `ADAPTERS` in `src/adapters.ts`: `exitCode` (above), `ready()` (your toolchain and build are present, including any plugin registration your runner needs, such as pytest's entry point) and `prepare()` (the command, args and env that launch your scenario, its `scenarioFile` (the spec your call sites must name, which is the copy when you run from one), plus an optional `cleanup`, and a `cwd` for a CLI that resolves its runner and plugins from the working directory, as WebdriverIO does). `tests/conformance.test.ts` then runs your adapter in all three BiDi modes, holds it to every rule, and compares it with every other adapter. Expect the first run to be red: classify each line as a suite assumption to fix, a difference to declare with a reason, or an adapter bug to fix, and merge only when it is green. `ready()` may only skip a local run; in CI a missing prerequisite fails.
 4. Make sure CI installs your toolchain before `pnpm test` in `.github/workflows/ci.yml` (Python's install step is there).
 
 Run it locally with `pnpm test packages/conformance`. Vitest only collects `packages/**/tests/**/*.test.ts`.
@@ -260,6 +260,9 @@ Defined in `src/checker.ts`, names in `RULES` (`src/constants.ts`):
 | `replace-targets-a-sent-row` | Every `replaceCommand.oldTimestamp` names a row already sent, by a `commands` frame or as an earlier replacement's stamp. |
 | `one-row-per-browser-action` | The number of `navigate`, `click` and `fill` rows equals what the fake browser received, after applying `replaceCommand` the way the dashboard does (shared `replaceCommand`: by `id`, then the last row with the old timestamp). |
 | `tests-end-as-designed` | Every test in `SCENARIO_TESTS` is in the stream's suite tree (the latest report of each), ending in the state the scenario designs: a passing test reported failed or skipped, or a test missing, breaks it. |
+| `every-test-has-rows` | Every scenario test the suite tree reports has at least one row. A row belongs to the test its `testUid` names, else to the last test started by then (`src/scenario-tests.ts`, the attribution the comparison uses). A test the tree never reports is left to `tests-end-as-designed`. |
+| `failed-test-has-error-row` | Every scenario test reported failed, or designed to fail whatever it is reported, has a row that is a failed action by shared's `commandFailure`. An unmapped command's error does not count: a wait's polls error whether the wait passes or not, and the trace drops them. |
+| `call-site-names-the-scenario` | Every row carrying a `callSource` names the spec the runner executed (`scenarioFile` on the adapter's launch, as launched or with symlinks resolved), at a line inside it: never DevTools' own code, another file, or the `unknown:0` placeholder. |
 | `run-controls-are-served` | The dashboard offers every control the runner's defaults allow unless `metadata.options.runCapabilities` turns it off (shared `resolveRunCapabilities`, the dashboard's own rule), so a stream that advertises any control, by declaration or by silence, serves it: `launchCommand` or `rerunCommand` in the metadata (the backend's shell path, falling back from either to the other), `configFile` there, or a `config` frame (its framework path, which reruns a WDIO or Nightwatch config). With none, the backend falls back to the WebdriverIO binary, so the control runs the wrong tool. |
 | `every-browser-request-listed` | For every url the browser loaded, the `networkRequests` rows (after folding re-sends under the same `id`, as the dashboard does) number at least the loads the browser made. Fewer means the page's Resource Timing was withheld although nothing else served the request. |
 | `one-row-per-browser-request` | For every url the browser loaded, the rows number at most the loads the browser made. More means two sources listed one load, or a re-sent row carried a new id. The two rules are separate so a declared gap can cover the first without excusing the second. |

packages/app/src/components/workbench/call-source.ts

@@ -4,24 +4,9 @@
  *  parsed stack traces. The format is identical across every runner/framework
  *  (WebdriverIO, Nightwatch, Selenium), so nothing here branches on framework. */
 
-/** Split a call source into its file and 1-based line.
- *  Anchors the line/column at the end so Windows drive paths (`C:\a.ts:10`)
- *  survive. Returns null when there's no trailing line number. */
-export function parseCallSource(
-  callSource: string
-): { file: string; line: number } | null {
-  // Match only the trailing line, with an optional column. Two flat patterns
-  // (no nested quantifier) keep it linear and handle Windows drive paths
-  // (`C:\a.ts:10`): everything before the matched `:line` is the file.
-  const match = callSource.match(/:(\d+):\d+$/) || callSource.match(/:(\d+)$/)
-  if (!match || match.index === undefined) {
-    return null
-  }
-  return {
-    file: callSource.slice(0, match.index),
-    line: parseInt(match[1], 10)
-  }
-}
+import { parseCallSource } from '@wdio/devtools-shared'
+
+export { parseCallSource }
 
 /** Path with any trailing `:line` / `:line:column` suffix stripped — some
  *  recorded traces glue the line onto the file, so lookups compare clean paths. */

packages/app/tests/call-source.test.ts

@@ -1,41 +1,12 @@
 import { describe, expect, it } from 'vitest'
 import {
-  parseCallSource,
   fileBasename,
   pathSegments,
   normalizeSourcePath,
   resolveSourceFile,
   listSourceFiles
 } from '../src/components/workbench/call-source.js'
 
-describe('parseCallSource', () => {
-  it('parses file:line', () => {
-    expect(parseCallSource('/a/b/login.e2e.ts:24')).toEqual({
-      file: '/a/b/login.e2e.ts',
-      line: 24
-    })
-  })
-
-  it('parses file:line:column, keeping only the line', () => {
-    expect(parseCallSource('/a/b/login.e2e.ts:24:11')).toEqual({
-      file: '/a/b/login.e2e.ts',
-      line: 24
-    })
-  })
-
-  it('tolerates Windows drive paths', () => {
-    expect(parseCallSource('C:\\tests\\login.ts:42:5')).toEqual({
-      file: 'C:\\tests\\login.ts',
-      line: 42
-    })
-  })
-
-  it('returns null without a trailing line number', () => {
-    expect(parseCallSource('/a/b/login.e2e.ts')).toBeNull()
-    expect(parseCallSource('')).toBeNull()
-  })
-})
-
 describe('fileBasename', () => {
   it('returns the last POSIX segment', () => {
     expect(fileBasename('/a/b/login.page.ts')).toBe('login.page.ts')

packages/conformance/src/adapters.ts

@@ -34,6 +34,9 @@ const MOCHA_BIN = path.join(
   'packages/conformance/node_modules/.bin/mocha'
 )
 
+const WDIO_SPEC = scenario('wdio/specs/scenario.e2e.js')
+const SELENIUM_SPEC = scenario('selenium/scenario.spec.js')
+
 /** Every adapter the suite runs, each through its framework's own runner. Adding one is an entry here plus its scenario folder. */
 export const ADAPTERS: AdapterScenario[] = [
   {
@@ -48,6 +51,7 @@ export const ADAPTERS: AdapterScenario[] = [
       ].every((f) => existsSync(f)),
     // From this package: the CLI resolves its runner, framework and service from the working directory.
     prepare: () => ({
+      scenarioFile: WDIO_SPEC,
       command: WDIO_BIN,
       args: ['run', scenario('wdio/wdio.conf.js')],
       cwd: path.join(REPO_ROOT, 'packages/conformance')
@@ -65,13 +69,14 @@ export const ADAPTERS: AdapterScenario[] = [
       ].every((f) => existsSync(f)),
     // From this package, so `--require` resolves the adapter as a user's project would.
     prepare: () => ({
+      scenarioFile: SELENIUM_SPEC,
       command: MOCHA_BIN,
       args: [
         '--require',
         '@wdio/selenium-devtools',
         '--timeout',
         '30000',
-        scenario('selenium/scenario.spec.js')
+        SELENIUM_SPEC
       ],
       cwd: path.join(REPO_ROOT, 'packages/conformance')
     })
@@ -87,6 +92,7 @@ export const ADAPTERS: AdapterScenario[] = [
         COLLECTOR_BUNDLE
       ].every((f) => existsSync(f)),
     prepare: () => ({
+      scenarioFile: scenario('nightwatch/specs/scenario.js'),
       command: NIGHTWATCH_BIN,
       args: ['-c', scenario('nightwatch/nightwatch.conf.cjs')]
     })
@@ -105,6 +111,7 @@ export const ADAPTERS: AdapterScenario[] = [
       const spec = path.join(dir, 'test_scenario.py')
       copyFileSync(scenario('python/test_scenario.py'), spec)
       return {
+        scenarioFile: spec,
         command: PYTHON,
         args: ['-m', 'pytest', '--devtools', '-p', 'no:cacheprovider', spec],
         cwd: dir,

packages/conformance/src/allowlist.ts

@@ -5,7 +5,7 @@ import {
   KNOWN_DEBT,
   RULES
 } from './constants.js'
-import { anyOf, hunk, reportedAs } from './gap-matchers.js'
+import { anyOf, failureUnexplained, hunk, reportedAs } from './gap-matchers.js'
 import { scenarioIndex } from './scenario-tests.js'
 import type {
   AdapterId,
@@ -75,6 +75,14 @@ export const KNOWN_GAPS: KnownGap[] = [
 function nightwatchGaps(bidi: BidiMode): KnownGap[] {
   const adapter = 'nightwatch-devtools'
   return [
+    {
+      adapter,
+      bidi,
+      rules: [RULES.failedTestExplained],
+      only: failureUnexplained(HEADING),
+      reason: KNOWN_DEBT.nightwatchSameMsReplace,
+      timingDependent: true
+    },
     {
       adapter,
       bidi,
@@ -183,6 +191,13 @@ function pythonGaps(bidi: BidiMode): KnownGap[] {
       lacksSteps: ['Element.waitForExist'],
       reason: KNOWN_DEBT.pythonWait
     },
+    {
+      adapter,
+      bidi,
+      rules: [RULES.failedTestExplained],
+      only: failureUnexplained(MISSING),
+      reason: KNOWN_DEBT.pythonWait
+    },
     {
       adapter,
       bidi,

packages/conformance/src/call-site.ts

@@ -0,0 +1,59 @@
+import path from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+import { parseCallSource, type CallSite } from '@wdio/devtools-shared'
+
+import { REPO_ROOT, RULES } from './constants.js'
+import type { RecordedStream, ScenarioFile, Violation } from './types.js'
+
+const OWN_PACKAGES = path.join(REPO_ROOT, 'packages') + path.sep
+
+/** Every row that carries a call site must name a line of the spec that ran, never DevTools' own code. */
+export function checkCallSites(
+  { rows }: RecordedStream,
+  scenario: ScenarioFile
+): Violation[] {
+  return rows.flatMap((row): Violation[] => {
+    if (!row.callSource) {
+      return []
+    }
+    const site = parseCallSite(row.callSource)
+    if (
+      site &&
+      scenario.paths.includes(site.file) &&
+      site.line >= 1 &&
+      site.line <= scenario.lines
+    ) {
+      return []
+    }
+    return [
+      {
+        rule: RULES.callSiteInScenario,
+        message: `${row.command} names ${row.callSource}: ${describeMiss(site, scenario)}`,
+        subject: { kind: 'row', timestamp: row.timestamp }
+      }
+    ]
+  })
+}
+
+function describeMiss(site: CallSite | undefined, scenario: ScenarioFile) {
+  if (!site) {
+    return 'not a file and line'
+  }
+  if (scenario.paths.includes(site.file)) {
+    return `the spec has ${scenario.lines} lines`
+  }
+  return site.file.startsWith(OWN_PACKAGES)
+    ? "DevTools' own code, not the spec"
+    : `a file other than the spec (${scenario.paths[0]})`
+}
+
+/** The dashboard's own reading of a call site, its file compared as a path. */
+function parseCallSite(callSource: string): CallSite | undefined {
+  const site = parseCallSource(callSource)
+  if (!site) {
+    return undefined
+  }
+  const { file, line } = site
+  return { file: file.startsWith('file://') ? fileURLToPath(file) : file, line }
+}

packages/conformance/src/capabilities.ts

@@ -32,6 +32,8 @@ export const INTERNAL_RULES: Partial<Record<RuleId, string>> = {
     'Checks that metadata reaches the dashboard before the first command, not the session, capabilities and device the Metadata tab shows.',
   [RULES.testsAsDesigned]:
     'Checks that each scenario test is reported in the state it was written to end in, a precondition of every per-test view rather than a feature.',
+  [RULES.testHasRows]:
+    'Checks that every reported test owns at least one row, a precondition of every per-test view rather than a feature.',
   [RULES.runControlsServed]:
     'Checks that every run control the dashboard offers has a command to serve it, not that Run, Rerun and Run all work.'
 }

packages/conformance/src/checker.ts

@@ -14,6 +14,8 @@ import {
 } from '@wdio/devtools-shared'
 
 import { ACTION_METHODS, RULES, SCOPES } from './constants.js'
+import { checkCallSites } from './call-site.js'
+import { checkTestRows } from './scenario-tests.js'
 import type {
   ActionMethod,
   GroundTruth,
@@ -38,7 +40,9 @@ export function checkConformance(
     ...rowPerAction(stream, truth),
     ...runControlsServed(stream),
     ...requestsListed(stream, truth),
-    ...requestSourceServed(stream, truth)
+    ...requestSourceServed(stream, truth),
+    ...checkCallSites(stream, truth.scenarioFile),
+    ...checkTestRows(stream)
   ]
 }
 

packages/conformance/src/constants.ts

@@ -41,6 +41,9 @@ export const RULES = {
   rowPerAction: 'one-row-per-browser-action',
   runControlsServed: 'run-controls-are-served',
   testsAsDesigned: 'tests-end-as-designed',
+  failedTestExplained: 'failed-test-has-error-row',
+  callSiteInScenario: 'call-site-names-the-scenario',
+  testHasRows: 'every-test-has-rows',
   agreeOnActions: 'adapters-agree-on-actions',
   agreeOnErrors: 'adapters-agree-on-errors',
   agreeOnPages: 'adapters-agree-on-pages',

packages/conformance/src/fake-webdriver.ts

@@ -30,7 +30,7 @@ import type {
   BidiMode,
   BrowserRequest,
   FailedNavigation,
-  GroundTruth,
+  BrowserTruth,
   PageDocument
 } from './types.js'
 
@@ -104,7 +104,7 @@ export class FakeWebDriver {
     return `${this.origin}${path}`
   }
 
-  truth(): GroundTruth {
+  truth(): BrowserTruth {
     return {
       documents: this.#documents.map(({ url, bornAt }) => ({ url, bornAt })),
       failedNavigations: [...this.#failedNavigations],

packages/conformance/src/features.ts

@@ -14,6 +14,7 @@ import {
   tested,
   unsupported
 } from './capability-helpers.js'
+import { RULES } from './constants.js'
 import type { Feature } from './types.js'
 
 const WDIO = ADAPTER_TESTS['devtools-service']
@@ -271,12 +272,7 @@ export const FEATURES = [
     id: 'source-tab',
     name: 'Source tab opens the test line that ran the command',
     tables: ['readme-dashboard', 'cross-framework'],
-    cells: {
-      'devtools-service': covered(`${WDIO}session.test.ts`),
-      'selenium-devtools': covered(`${SELENIUM}index.test.ts`),
-      'nightwatch-devtools': covered(`${NIGHTWATCH}browserProxy.test.ts`),
-      'selenium-devtools-py': covered(`${PYTHON}test_pytest_plugin.py`)
-    }
+    cells: allTested([RULES.callSiteInScenario])
   },
   {
     id: 'metadata-tab',
@@ -306,7 +302,13 @@ export const FEATURES = [
     id: 'failed-command-errors',
     name: "Failed command in the Errors tab, with the browser's message",
     tables: ['readme-dashboard', 'cross-framework'],
-    cells: allTested(ERROR_RULES)
+    // Nightwatch and Python hold gaps on a failed test's error row (#458, #461), so only the others are tested on it.
+    cells: {
+      'devtools-service': tested([...ERROR_RULES, RULES.failedTestExplained]),
+      'selenium-devtools': tested([...ERROR_RULES, RULES.failedTestExplained]),
+      'nightwatch-devtools': tested(ERROR_RULES),
+      'selenium-devtools-py': tested(ERROR_RULES)
+    }
   },
   {
     id: 'run-controls',

packages/conformance/src/gap-matchers.ts

@@ -55,6 +55,12 @@ export function reportedAs(test: number, state: string): MessageMatcher {
   return (message) => message.startsWith(lead)
 }
 
+/** A failed scenario test none of whose rows is a failed action. */
+export function failureUnexplained(test: number): MessageMatcher {
+  const lead = `"${SCENARIO_TESTS[test].title}" failed but`
+  return (message) => message.startsWith(lead)
+}
+
 export const anyOf =
   (...matchers: MessageMatcher[]): MessageMatcher =>
   (message) =>