feat(conformance): generate the adapter capability matrix from one declared source

.changeset/nightwatch-run-all.md

@@ -0,0 +1,7 @@
+---
+'@wdio/nightwatch-devtools': patch
+'@wdio/devtools-backend': patch
+'@wdio/devtools-app': patch
+---
+
+Enable Run all for Nightwatch, which the dashboard disabled although the backend could already run the whole config. Run all now replays the selection the run was launched with (spec paths, `--tag`, `--group`, `--env` and the like) rather than every spec in the config, and a rerun in between does not narrow it.

.changeset/selenium-run-all-launch-command.md

@@ -0,0 +1,5 @@
+---
+'@wdio/selenium-devtools': patch
+---
+
+Keep Run all on the original command after a targeted rerun. A rerun's process was started with a name filter, and Run all used to be built from that process's own arguments, so it replayed the filter; it now inherits the command the dashboard's first run published. Run all also no longer comes out as `npm run npx` when the tests were started with `npx` or `pnpm exec`, and a spec path or filter containing a space now reaches the relaunched run as one argument.

.github/pull_request_template.md

@@ -29,6 +29,10 @@
 - [ ] `script` (page-injected runtime)
 - [ ] `trace` (Trace mode)
 
+## Capability matrix
+
+- [ ] `packages/conformance/src/features.ts` updated, and tables regenerated with `pnpm docs:capabilities`, if this adds or changes user-facing behaviour
+
 ## Notes for reviewers
 
 [//]: # (Anything non-obvious: design trade-offs, alternatives you ruled out, follow-ups. If this PR touches more than one adapter, say why the shared logic doesn't live in `core`.)

CONTRIBUTING.md

@@ -107,6 +107,16 @@ Python CI run fails on *"Pinned backend serves this adapter's contract"*, the
 answer is that the backend has not been released yet, not that your branch is
 wrong. Raise it by hand only when you are also releasing the backend.
 
+## Adding a feature
+
+Every user-facing feature is listed in [`packages/conformance/src/features.ts`](./packages/conformance/src/features.ts), with what each of the four adapters (WebdriverIO, Selenium, Nightwatch, Python) does with it. The capability tables in this README and on the docs site are generated from it.
+
+1. Add an entry: an id, a name, the tables it appears in, and a cell for every adapter.
+2. A supported cell names its evidence with the helpers in `capability-helpers.ts` beside it: `tested(rules)` for conformance rules that check it, `covered(path)` for a test file in that adapter's own tests (or `core`, `trace` or `backend` when the adapter adds no logic of its own), or `byHand(reason)` saying how to check it by hand. A partial or unsupported cell names the reason, which becomes the table's footnote.
+3. Run `pnpm docs:capabilities` to regenerate the README tables. `pnpm docs:capabilities --parity` lists every feature some adapter lacks.
+
+CI fails when an adapter is missing from an entry, a reason or evidence is missing, a test file does not exist or sits outside those tests, a new option or conformance rule maps to no feature, or the README tables are out of date.
+
 ## Before you push
 
 - `pnpm build`, `pnpm test`, and `pnpm lint` all green — don't push red.

README.md

@@ -40,6 +40,47 @@ Want a portable artifact instead of a live UI (CI / agent diffing)? Switch to **
 - **Jasmine**: Complete integration with grep-based filtering
 - **Cucumber**: Scenario-level and example-specific execution with feature:line targeting
 
+**Dashboard and options by adapter:**
+
+<!-- capability-matrix:readme-dashboard:start -->
+<!-- Generated by `pnpm docs:capabilities` in webdriverio/devtools from packages/conformance/src/features.ts. Do not edit by hand. -->
+
+| Feature | WebdriverIO | Selenium | Nightwatch | Python |
+|---|---|---|---|---|
+| One row per command in the Actions list | ✅ tested | ✅ tested | ✅ tested | ✅ tested |
+| Screenshot on every command row | ✅ | ✅ | ✅ | ✅ |
+| Browser console logs | ✅ | ✅ | ✅ | ✅ |
+| Source tab opens the test line that ran the command | ✅ | ✅ | ✅ | ✅ |
+| Metadata tab (session, capabilities, device) | ✅ | ✅ | ✅ | ✅ |
+| Performance timings on navigation rows | ✅ | ✅ | ✅ | ✅ |
+| Failed command in the Errors tab, with the browser's message | ✅ tested | ✅ tested | ✅ tested | ✅ tested |
+| Run, Rerun and Run all from the dashboard | ⚠️ no Cucumber step rerun ¹ | ⚠️ no Cucumber step rerun ¹ | ⚠️ no Cucumber step rerun ¹ | ✅ |
+| Preserve & Rerun (compare a failed run with its rerun) | ✅ | ✅ | ✅ | ✅ |
+| Dashboard window opens by itself in live mode, and can be turned off | ⚠️ always opens ² | ✅ `openUi: false` | ⚠️ always opens ³ | ✅ `DEVTOOLS_OPEN=0` |
+| Turn off per-command screenshots (`captureScreenshots`) | ❌ ⁴ | ✅ | ❌ ⁵ | ❌ ⁵ |
+| Force the test browser headless (`headless`) | ❌ ⁶ | ⚠️ Chrome only ⁷ | ❌ ⁶ | ❌ ⁶ |
+| Safari kept frontmost so input lands (live mode) | ⚠️ once ⁸ | ✅ before each element input | ⚠️ once ⁸ | ⚠️ once ⁸ |
+
+✅ tested: proven for that adapter by the cross-adapter conformance suite. ✅: supported, backed by a named test or a manual check listed in [the feature list](https://github.com/webdriverio/devtools/blob/main/packages/conformance/src/features.ts).
+
+¹ Under Cucumber a Gherkin step has no CLI filter of its own, so the dashboard reruns scenarios and features, not single steps.
+
+² The service always opens the dashboard in live mode; `devtoolsCapabilities` picks its browser but cannot skip it.
+
+³ Nightwatch always opens the dashboard in live mode.
+
+⁴ No option: every command row takes a screenshot, except on Appium, where only native sessions in live mode do.
+
+⁵ No option: every command row takes a screenshot.
+
+⁶ No option: set headless in the browser capabilities, as for any run.
+
+⁷ The option adds `--headless=old` to `goog:chromeOptions` only, so other browsers stay headed; set their own headless capability.
+
+⁸ Brings Safari to the front once after the session starts (live mode, local macOS), so a click issued after another window took the front can still land nowhere.
+
+<!-- capability-matrix:readme-dashboard:end -->
+
 ### 📊 Real-Time Visualization
 - **Live Browser Preview**: View the application under test in a scaled iframe with automatic screenshot updates
 - **Actions Timeline**: Command-by-command execution log with timestamps and parameters
@@ -102,29 +143,72 @@ The dashboard runs in one of **two modes**, set per adapter via the shared `mode
 
 All three JavaScript adapters (`@wdio/devtools-service`, `@wdio/selenium-devtools`, `@wdio/nightwatch-devtools`) emit the **same normalized trace** through the shared `@wdio/devtools-core` capture library, so one archive format and one player serve every framework.
 
-The trace **format and the player are identical** across those three, but **capture completeness varies** — WebdriverIO is the most complete; Selenium and Nightwatch cover the core flow with some gaps (e.g. inline-Allure per-test artifacts, retry-aware retention, Cucumber step nesting, auto BiDi). Each adapter's README lists its specifics.
+The Python adapter (`selenium-devtools-py`) writes the same archive through the backend, so the trace **format and the player are identical** across all four, but **capture completeness varies**: WebdriverIO is the most complete, and the others cover the core flow with the gaps the table below names. Each adapter's README lists its specifics.
 
-**Trace-mode support by adapter:**
+**Support by adapter:**
 
-| Capability | WebdriverIO | Selenium | Nightwatch |
-|---|---|---|---|
-| Trace mode + `show-trace` player | ✅ | ✅ | ✅ |
-| DOM time-travel (mutation capture) | ✅ | ✅ ¹ | ✅ |
-| Dense filmstrip (`filmstrip`) | ✅ CDP push | ✅ CDP push | ⚠️ polling only |
-| Per-test granularity (`traceGranularity: 'test'`) | ✅ | ✅ | ⚠️ ² |
-| Retry-aware retention (`tracePolicy`) | ✅ | ✅ | ⚠️ ³ |
-| Cucumber step nesting | Scenario→Step ⁴ | ✅ full | Feature→Scenario ⁵ |
-| Inline Allure attach | ✅ | ✅ | ⚠️ produce-only ⁶ |
-| Assertion capture (`captureAssertions`) | ✅ | ✅ | ✅ |
-| Auto BiDi capture | ✅ auto | ✅ auto | ⚠️ opt-in ⁷ |
-
-¹ With BiDi, Selenium registers the page collector at document start, so every document anchors its own DOM as WebdriverIO's does. Without BiDi it falls back to injecting the collector after each navigation, and there a navigation's snapshot can lag the command that triggered it.
-² Nightwatch's Cucumber and exports-object interfaces get real per-test slicing; the BDD `describe/it` interface collapses to a single session-scoped slice keyed to the first test.
-³ Only `retain-on-failure` works; the other retry-aware policies degrade to it because Nightwatch's `--retries` re-runs a testcase internally without re-firing the per-test hooks.
-⁴ WebdriverIO does not yet carry feature-level ancestry, so its Cucumber nesting is Scenario→Step.
-⁵ Nightwatch does not yet stamp per-step nesting (Feature→Scenario only).
-⁶ Nightwatch has no live Allure attach API, so per-test `screenshot`/`video` are written to disk and listed in the manifest but not attached to an Allure test.
-⁷ Opt-in via `bidi: true` + `webSocketUrl: true` in capabilities.
+<!-- capability-matrix:readme-trace:start -->
+<!-- Generated by `pnpm docs:capabilities` in webdriverio/devtools from packages/conformance/src/features.ts. Do not edit by hand. -->
+
+| Feature | WebdriverIO | Selenium | Nightwatch | Python |
+|---|---|---|---|---|
+| Trace mode + `show-trace` player | ✅ | ✅ | ✅ | ✅ |
+| DOM time-travel (mutation capture) | ✅ tested | ⚠️ with BiDi ¹ | ⚠️ with BiDi ¹ | ✅ tested |
+| Network requests (browser sessions) | ✅ tested | ⚠️ with BiDi ² | ⚠️ with BiDi ³ | ✅ tested |
+| Screencast (`filmstrip` / video) | ✅ CDP push (Chrome), polling elsewhere | ✅ CDP push (Chromium), polling elsewhere | ⚠️ polling only ⁴ | ✅ CDP push (Chromium), one frame per command elsewhere |
+| Per-test granularity (`traceGranularity: 'test'`) | ✅ | ✅ | ⚠️ ⁵ | ✅ |
+| Retry-aware retention (`tracePolicy`) | ✅ | ✅ | ⚠️ `retain-on-failure` only ⁶ | ⚠️ `retain-on-failure` only ⁷ |
+| Per-test `screenshot` / `video`, attached to Allure | ✅ | ✅ | ⚠️ produce-only ⁸ | ❌ ⁹ |
+| Cucumber Feature→Scenario→Step nesting | ⚠️ Scenario→Step ¹⁰ | ✅ | ⚠️ Feature→Scenario ¹¹ | ❌ ¹² |
+| Assertion capture (`captureAssertions`) | ✅ | ⚠️ `node:assert` only ¹³ | ⚠️ no `expect` ¹⁴ | ✅ |
+| BiDi console / network capture | ✅ auto | ✅ auto | ⚠️ opt-in ¹⁵ | ✅ auto |
+| A11y tab + pick-locator overlay (trace player) | ✅ | ✅ | ✅ | ✅ |
+| Transcript + Copy-for-LLM (trace player) | ✅ | ✅ | ✅ | ✅ |
+| Unpacked trace output (`traceFormat: 'ndjson-directory'`) | ✅ | ✅ | ✅ | ❌ zip only ¹⁶ |
+| Artifacts manifest for CI and reporters (`emitArtifactsManifest`) | ✅ | ✅ | ⚠️ opt-in ¹⁷ | ❌ ¹⁸ |
+| Network without BiDi (Safari, Appium mobile browsers) | ⚠️ ¹⁹ | ⚠️ ¹⁹ | ⚠️ ¹⁹ | ⚠️ ¹⁹ |
+
+✅ tested: proven for that adapter by the cross-adapter conformance suite. ✅: supported, backed by a named test or a manual check listed in [the feature list](https://github.com/webdriverio/devtools/blob/main/packages/conformance/src/features.ts).
+
+¹ Without BiDi (Safari, Appium) the page collector is injected after each navigation, so a page the test leaves within tens of milliseconds is never captured.
+
+² Without BiDi a page the test leaves within tens of milliseconds is never drained, so its requests are not listed.
+
+³ Without BiDi (`bidi: true`) or Chrome's performance log (`goog:loggingPrefs`), a page the test leaves within tens of milliseconds is never drained, so its requests are not listed.
+
+⁴ Nightwatch exposes no stable CDP handle, so frames come from a `takeScreenshot` poll every 200 ms.
+
+⁵ Nightwatch fires the global `beforeEach` / `afterEach` once per test module for `describe/it` and exports-object tests, so they collapse to one slice keyed to the first test; Cucumber scenarios get real per-test slices.
+
+⁶ For `describe/it` and exports-object tests, `--retries` reruns a testcase internally without firing the per-test hooks again, so the other policies behave like `retain-on-failure`; Cucumber scenarios record every attempt.
+
+⁷ Nothing the Python adapter sends carries an attempt number, so the retry-aware policies behave like `retain-on-failure`.
+
+⁸ Nightwatch has no live Allure attach API, so per-test screenshots and videos are written to disk and listed in the manifest but not attached.
+
+⁹ The Python adapter has no `screenshot` / `video` options and no Allure attachment yet.
+
+¹⁰ The service records no feature ancestry for a scenario, so the trace nests Scenario→Step.
+
+¹¹ Nightwatch steps are not stamped as nested groups, so the trace nests Feature→Scenario.
+
+¹² The Python adapter stamps no Feature or Step groups, so a pytest-bdd scenario is one flat test group with no Feature parent and no Step children.
+
+¹³ `node:assert` rows pass and fail; jest-style `expect()` and chai produce no rows at all, and a failing matcher only marks its test failed.
+
+¹⁴ `assert.*`, `verify.*` and `node:assert` rows pass and fail; Nightwatch's `expect` API produces no rows.
+
+¹⁵ Nightwatch turns it on with `bidi: true` plus `webSocketUrl: true` in the capabilities.
+
+¹⁶ The Python adapter has no option for it and sends no format in its export request, so the backend writes a zip.
+
+¹⁷ Nightwatch has no live Allure signal to turn the manifest on by itself, so it stays opt-in.
+
+¹⁸ The Python adapter writes no artifacts manifest.
+
+¹⁹ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari.
+
+<!-- capability-matrix:readme-trace:end -->
 
 In trace mode no DevTools UI window opens. At session end the adapter writes trace artifacts into a `test-results/` folder (created next to the resolved spec/config directory), suitable for offline replay, AI-agent diffing, or any consumer that prefers a portable artifact over a live UI.
 
@@ -133,6 +217,7 @@ In trace mode no DevTools UI window opens. At session end the adapter writes tra
 | **WebdriverIO** | `services: [['devtools', { mode: 'trace' }]]` |
 | **Selenium** | `DevTools.configure({ mode: 'trace' })` (before importing `selenium-webdriver`) |
 | **Nightwatch** | `globals: nightwatchDevtools({ mode: 'trace' })` |
+| **Python** | `pytest --devtools-trace`, or `devtools.enable(trace=True)` in a plain script |
 
 The trace artifact contains:
 - `trace.trace` — NDJSON `context-options` + `before`/`after` action events. When test hooks are available (Mocha's `it()` / Cucumber's `Scenario()`), each test becomes a `Tracing.tracingGroup` span — an open/close `before`/`after` pair with `method: "tracingGroup"` and `params.name` set to the test title. Child actions inside the group carry `parentId` pointing back to the group's `callId`, so timeline viewers render tests as labelled spans wrapping their commands.
@@ -216,7 +301,29 @@ The discriminator is the **browser name in the capabilities**, not the device: m
 
 A hybrid app is the case capabilities alone cannot answer, because the answer changes mid-run. Appium reports `NATIVE_APP` or a webview context, and anything that is not `NATIVE_APP` counts as a webview — the `WEBVIEW_` prefix is a convention, and a driver naming its webview otherwise would have its DOM capture skipped. Following the context costs no round trip: the switch command carries its destination in its own arguments.
 
-**Per-adapter:** all four adapters detect a native session and skip page-side capture on one. Only the **WebdriverIO** service additionally follows a hybrid app into its webview; Nightwatch, Selenium and the Python adapter answer from the startup capabilities, so a hybrid session's webview half is captured as if it were still native.
+**Per-adapter:**
+
+<!-- capability-matrix:readme-mobile:start -->
+<!-- Generated by `pnpm docs:capabilities` in webdriverio/devtools from packages/conformance/src/features.ts. Do not edit by hand. -->
+
+| Feature | WebdriverIO | Selenium | Nightwatch | Python |
+|---|---|---|---|---|
+| Network without BiDi (Safari, Appium mobile browsers) | ⚠️ ¹ | ⚠️ ¹ | ⚠️ ¹ | ⚠️ ¹ |
+| Mobile browser sessions (Chrome on Android, Safari on iOS) | ✅ | ✅ | ✅ | ✅ |
+| iOS (XCUITest): native apps and Safari | ✅ | ✅ | ✅ | ✅ |
+| Native session detected, page-side capture skipped | ✅ | ✅ | ✅ | ✅ |
+| Follows a hybrid app into its webview | ⚠️ string form only ² | ⚠️ ³ | ⚠️ ³ | ⚠️ ³ |
+| Native app network through appium-interceptor (`appium:intercept`) | ✅ Android | ✅ Android | ✅ Android | ✅ Android |
+
+✅ tested: proven for that adapter by the cross-adapter conformance suite. ✅: supported, backed by a named test or a manual check listed in [the feature list](https://github.com/webdriverio/devtools/blob/main/packages/conformance/src/features.ts).
+
+¹ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari.
+
+² Follows `switchContext('WEBVIEW_...')`; the object form (`{ title }` / `{ url }`) and `switchAppiumContext` leave capture on the native side.
+
+³ Decides from the startup capabilities, so a hybrid app's webview half is captured as if it were still native.
+
+<!-- capability-matrix:readme-mobile:end -->
 
 **In the dashboard**, a phone capture is framed as the device — a full-height device column with the action list and dock beside it — in both live mode and the trace player. The trace's `context-options` records the platform, model and OS version, read back so the player labels frames without guessing.
 

package.json

@@ -24,6 +24,7 @@
     "demo:python:mobile": "python3 examples/selenium-py/mobile/${DEVTOOLS_MOBILE_PLATFORM:-android}/*.py",
     "dev": "pnpm --parallel dev",
     "preview": "pnpm --parallel preview",
+    "docs:capabilities": "tsx packages/conformance/src/capability-matrix.ts",
     "test": "vitest run",
     "test:watch": "vitest",
     "test:coverage": "vitest run --coverage",