[vishnuv688]

What & why

Closes #402

The adapter capability tables were maintained by hand in three places (the README, CrossFrameworkSupport.md and MobileTesting.md on the docs site). They disagreed with each other, and two of them had no Python column, because a markdown table cannot notice that a fourth adapter exists.

The matrix is now an output. packages/conformance/src/features.ts lists every user-facing feature once, with a cell for each of the four adapters (WebdriverIO, Selenium, Nightwatch, Python), and pnpm docs:capabilities renders the tables from it. The conformance package holds the helpers, validation and rendering.

• Every adapter, every row. Cells are typed as a full map over the conformance suite's adapter ids, so a fifth adapter fails typecheck until every row says what it does.
• A reason is required. A partial or unsupported cell without a reason is a type error. Reasons render as footnotes, written from the code and the Known debt entries in CLAUDE.md.
• Evidence on every supported cell. Each one names how it is known to hold: conformance rules (rendered "✅ tested"), a test file in that adapter's or a shared package's tests (validation checks it is a file in an allowed folder), or how to check it by hand. A rule must exist and the adapter must have no known gap against it.
• Nothing can skip the matrix.
  • Every user-facing option must map to a capability row or say why it is not one. That covers the shared base, the WDIO service, Selenium and Nightwatch option types (checked at compile time against the real types) and Python's enable() keywords, pytest options and DEVTOOLS_* variables (checked by a test that parses the source with Python's ast).
  • Every conformance rule must back a tested cell or be listed as internal with a reason.
• Parity. pnpm docs:capabilities --parity lists every feature some adapter lacks, with the reason, and counts cells by evidence kind.
• Rendering. pnpm docs:capabilities rewrites the content between <!-- capability-matrix:<table>:start --> and :end --> markers. The README now holds three generated tables: trace mode, live dashboard and mobile. --check writes nothing and exits 1 when anything is stale, and a test runs it against the README. --website <checkout> also rewrites the two docs-site pages; it checks every target first and writes nothing when a page is missing its markers.

The list has 33 features. Each adapter's cells were checked against its code by a separate pass, and every test link was audited; links that did not assert their feature now name one that does or a manual check. Today: 18 of 33 features are not supported by every adapter; of 132 cells, 12 are conformance-tested, 57 unit-tested, 22 checked by hand and 41 partial or unsupported. Correcting the old tables along the way:

• Python is in every table.
• Selenium and Nightwatch DOM time-travel and network are "with BiDi", where the README showed full support. Without BiDi the suite records known gaps for both.
• Selenium assertion capture is partial (jest and chai expect() produce no rows), Selenium headless is Chrome only, and Selenium Run all can narrow to a rerun's filter.
• Nightwatch: Run all is disabled because the adapter declares canRunAll: false (the backend already supports it, so the conformance gap's reason is corrected too); only Cucumber gets per-test slices; the expect API is not captured.
• WDIO follows a hybrid webview only through switchContext('WEBVIEW_...'). Screencast push is Chromium only for every adapter.
• Cucumber nesting cells such as "Scenario→Step" carried no status mark and now render as partial with reasons.
• New rows include network, failed commands, run controls, Preserve & Rerun, the dashboard and screenshot switches, Safari focus, network without BiDi, iOS and mobile browsers.

The PR template gains a feature-list checkbox, CONTRIBUTING.md an "Adding a feature" section, and CLAUDE.md documents the command and both coverage checks. Two comment-only fixes ride along in published packages (Selenium's headless docstring, the service's Cucumber nesting comment), so no changeset.

Run all for Nightwatch and Selenium

The per-adapter checks found Run all broken in two adapters, fixed here because the matrix would otherwise declare the gap rather than close it:

• Nightwatch: the adapter and the shared defaults declared canRunAll: false although the backend already relaunches a Nightwatch config. Run all is now on and replays the selection the run was launched with (spec paths made absolute, flag values such as --tag smoke kept as typed, a stale selection cleared). The value is recorded by the run that starts the backend, encoded once in shared, and replayed by runAllFilters in framework-filters.ts. A rerun child attaches to that backend and cannot narrow it. The conformance suite's Nightwatch run-controls gap, and the gap machinery only it used, are removed.
• Selenium: Run all was rebuilt from the current process's arguments, so after a targeted rerun it replayed that rerun's filter. It now inherits the parent's launch command, as the Python adapter does. Under npx and pnpm exec (npm_command=exec) it no longer publishes npm run npx, and the argv fallback is shell-quoted so a path or filter with a space survives.
• Under Cucumber a single Gherkin step cannot be rerun in any adapter that runs Cucumber, so the run-controls cell shows that shared limit on WDIO, Selenium and Nightwatch.

Type of change

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Polish (an improvement to an existing feature)
• Breaking change (existing behavior changes for users)
• Documentation
• Internal (build, CI, dependencies, tooling)

Packages touched

• shared (types and contracts)
• core (framework-agnostic capture/reporting)
• elements (published element/snapshot API — @wdio/elements)
• service (WebdriverIO adapter)
• nightwatch-devtools (Nightwatch adapter)
• selenium-devtools (Selenium adapter)
• selenium-devtools-py (Selenium Python adapter)
• backend (server)
• app (UI)
• script (page-injected runtime)
• trace (Trace mode)

The code lives in packages/conformance, which is private and test-only, beside the adapter ids and rules it uses. It imports only the adapters' option types. Changesets: patch for @wdio/nightwatch-devtools, @wdio/devtools-backend and @wdio/devtools-app (Nightwatch Run all), and for @wdio/selenium-devtools (launch command).

Notes for reviewers

• Where it lives. All in packages/conformance/src: features.ts (the list alone), cell builders and the one "tested" predicate in capability-helpers.ts, validation in capability-validation.ts, option maps in capability-options.ts, rendering in render-capabilities.ts, the CLI in capability-matrix.ts.
• Tested cells are conservative. A cell is "tested" only where a conformance rule covers it and the adapter has no known gap against it. That is DOM time-travel and network for WDIO and Python, plus failed commands, run controls, the one-row-per-command rules and metadata. Everything else is declared.
• The internal rules are the six adapters-agree-* comparisons, which judge adapters against each other rather than any one adapter.
• Checks proven. Adding an unmapped option to Selenium's DevToolsOptions failed pnpm typecheck, and adding a keyword to Python's enable() failed the coverage test. Both were reverted.
• Docs site. The website pages need the markers inserted once. A follow-up webdriverio PR adds them and the generated tables, and rewrites the sentence on CrossFrameworkSupport.md that says Python is not in the table. Dry-run against a copy of the docs, the generator renders all 33 rows on that page and the 6 mobile rows on MobileTesting.md.
• Follow-ups found by the checks, not done here: a suspected duplicate navigation row in WDIO live mode (timings re-sent with sendUpstream instead of a replace); a stale Python docstring.
• Run all in a real browser is not verified yet. CI covers it through unit tests and the conformance suite. To check: Nightwatch narrowed launch (npx nightwatch -c ../../examples/nightwatch/nightwatch.conf.cjs ../../examples/nightwatch/bdd/smoke-test.js from packages/nightwatch-devtools), pnpm demo:nightwatch, the Nightwatch Cucumber example, pnpm --filter @wdio/selenium-devtools example:mocha with a rerun then Run all, and Selenium under npx mocha.
• Results. pnpm test: 2867 passed. pnpm lint: 0 errors and no new warnings. pnpm typecheck, pnpm build and pnpm docs:capabilities --check are clean.

Screenshots / recordings

n/a (no UI change)

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; the latest Nightwatch lookup change fixes the remaining reported failure.

Summary

The PR generates adapter capability tables from one feature list and preserves the original selection for Run all.

• One checked feature list now generates the adapter capability tables.
• Nightwatch Run all now replays the selection from the original launch.
• Selenium Run all now uses the original launch command after a rerun.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Dashboard rerun or Run all] --> B{Existing binary override?}
  B -->|Yes| C[Use override]
  B -->|No| D[Search config directory and parents]
  D --> E{Nightwatch found?}
  E -->|Yes| F[Use found binary]
  E -->|No| G[Search launch directory and parents]
  G --> H{Nightwatch found?}
  H -->|Yes| F
  H -->|No| I[Report missing binary]
  C --> J[Spawn from launch directory]
  F --> J

Loading

_{Reviews (9) · Last reviewed commit: "fix(backend): fall back to the launch di..." · Reviewed by Greptile}