Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/nightwatch-passing-test-without-assertions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@wdio/nightwatch-devtools': patch
---

A Nightwatch test that makes no Nightwatch assertion is reported as passed, and one with errors as failed.
1 change: 0 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,6 @@ Documented divergences from the conventions above. They exist today as debt to b
- **Two mechanisms resolve a command's target selector, and the WDIO one is wrong for interleaved handles.** Selenium keys a `WeakMap` on handle identity (`WebElement.id_` is a promise, so no id is readable when a command is invoked); the service uses a mutable last-selector in `service/src/command-selectors.ts`, which stamps the wrong locator for `const a = await $('#a'); const b = await $('#b'); await a.click()` (the Selenium side is covered in `selenium-devtools/tests/element-locators.test.ts`; the WDIO failure itself has **no** test — it is an unverified reading of `command-selectors.ts`). They cannot share a registry — WDIO's hook sees a *serialized* handle carrying an id string and never the live object, Selenium has the object and no readable id. Unifying would need the policy parameterized rather than the storage shared; until then the WDIO path is knowingly wrong in that case. Nightwatch is a third mechanism: it reads arg 0 through a per-kind allowlist (`assertTarget.ts`) rather than tracking handles at all, because its classic API takes selector strings. The allowlists are deliberately **not** derived from shared's `ACTION_MAP` — that table says how a command *renders* (its `Element` entries include WDIO commands called on a handle, with no selector argument), not whether arg 0 is an element definition.
- Service renders expect-webdriverio matchers as single `expect.<matcher>` rows by **folding**, not stack/depth suppression (the old `#assertionDepth`/`#matcherStarted`/self-heal machinery is gone). The matcher's value-read (`toHaveText`→`getText`, `toExist`→`isExisting`, …) is captured as a normal command; `afterAssertion` then coalesces the synthesized `expect.*` row into that read in place — inheriting its callSource, screenshot, and timeline position — and the fold replaces **by timestamp, never a public `id`**: `id` is the per-worker `commandCounter`, which resets per spec, so stamping one lets the app's id-first `replaceCommand` swap a same-id row from another spec (duplicate rows + a fold from another spec vanishing, in multi-spec live mode). `beforeAssertion` arms the pending matcher (depth-counted so aliases like `toBeChecked`→`toBeSelected` fold once); a matcher that **hard-throws** — element never resolves, so expect-webdriverio's `waitUntil` rethrows and `afterAssertion` never fires — is synthesized at `afterTest`/`afterStep` from the throwing read, so a failing assertion renders as `expect.<matcher>` whether or not the element existed. Two limits: its error is then the read's (`Can't call getText on … element wasn't found`), not an assertion-phrased message; and `MATCHER_READ_COMMANDS` is a hand-maintained allowlist, so a matcher whose read isn't listed leaves its raw read visible alongside the `expect.*` row. Plain-value jest matchers (`expect(x).toBe(y)`) don't fire the ewdio hooks, so they aren't captured as rows.
- **What the conformance scenario shows the adapters doing differently.** Each entry is a `KNOWN_GAPS` reason in `packages/conformance/src/allowlist.ts`, scoped to the differences it explains (one hunk, one moved step, or one test's reported state), and reported stale once they stop occurring, except an entry marked `timingDependent`, which is never reported stale and has to be removed by hand.
- **Nightwatch reports a passing test skipped** (#459). The scenario's two passing tests that call no assertion (the condition waits, and the caught refused navigation) run, their rows are in the stream, and the runner does not count them failed, but the suite tree reports both `skipped` in every run; the passing test that asserts is reported `passed`.
- **A Selenium or Python wait has no row when it succeeds, and a condition row when it fails** (#464). A successful `driver.wait(until.elementLocated(...))` or `WebDriverWait(...).until(EC.presence_of_element_located(...))` is a folded `findElements`/`findElement` row, which ACTION_MAP leaves unmapped; a timed-out one is a `wait` row named after the condition and mapped to `Browser.waitForFunction` without a target, where the WDIO service and Nightwatch show `Element.waitForExist` on the selector. Python reaches the Node.js adapter's shape (`selenium-devtools/src/wait-tracking.ts`) from `selenium_devtools/wait_tracking.py`, which patches `WebDriverWait.until`/`until_not`, because the wait is a client-side loop the `execute` chokepoint sees only as its polls; an `expected_conditions` closure is named after its factory and captured arguments, since its own name is `_predicate`.
- **A condition wait shows its reads in some adapters and one wait row in others** (#464). A passing wait on a condition that reads the page (`waitUntil(() => getText ...)`, `driver.wait(async () => ...)`, `WebDriverWait(...).until(lambda d: ...)`) is one `Browser.waitForFunction` row in the WebdriverIO service, which logs only the command on top of its stack and so hides the reads inside it; its reads and no wait row in Selenium and Python; and both in Nightwatch. The reads are what show a fold bug (one read folded over another), so the scenario keeps them visible where the adapter shows them.

Expand Down
14 changes: 3 additions & 11 deletions packages/conformance/src/allowlist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import {
KNOWN_DEBT,
RULES
} from './constants.js'
import { anyOf, hunk, reportedAs } from './gap-matchers.js'
import { anyOf, hunk } from './gap-matchers.js'
import { scenarioIndex } from './scenario-tests.js'
import type {
AdapterId,
Expand All @@ -16,10 +16,9 @@ import type {
Violation
} from './types.js'

const [WAITS, MISSING, REFUSED] = [
const [WAITS, MISSING] = [
'waits on what the page shows',
'times out waiting for a missing element',
'survives a refused navigation'
'times out waiting for a missing element'
].map(scenarioIndex)

const FLASH_WAIT = 'Element.waitForExist #flash'
Expand Down Expand Up @@ -72,13 +71,6 @@ export const KNOWN_GAPS: KnownGap[] = [
function nightwatchGaps(bidi: BidiMode): KnownGap[] {
const adapter = 'nightwatch-devtools'
return [
{
adapter,
bidi,
rules: [RULES.testsAsDesigned],
only: anyOf(reportedAs(WAITS, 'skipped'), reportedAs(REFUSED, 'skipped')),
reason: KNOWN_DEBT.nightwatchSkipped
},
{
adapter,
bidi,
Expand Down
1 change: 0 additions & 1 deletion packages/conformance/src/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,6 @@ const SELENIUM_WAIT_DEBT =

/** Why each scenario gap exists: the limitation, and the Known debt entry that describes it. */
export const KNOWN_DEBT = {
nightwatchSkipped: `Nightwatch reports a passing test that ran no assertion as skipped, although it ran. ${debt('Nightwatch reports a passing test skipped', '#459')}`,
conditionWait: `A passing condition wait shows as its reads in Selenium and Python, which have no row for the wait itself, as one Browser.waitForFunction row in the WebdriverIO service, which hides the reads inside it, and as both in Nightwatch. ${debt('A condition wait shows its reads in some adapters and one wait row in others', '#464')}`,
seleniumWait: `A successful driver.wait() has no row of its own: its polls are findElements rows, which the trace vocabulary leaves unmapped. ${debt(SELENIUM_WAIT_DEBT, '#464')}`,
seleniumFailedWait: `A timed-out driver.wait() is a wait row named after its condition, mapped to Browser.waitForFunction with no target, where the other adapters show Element.waitForExist on the selector. ${debt(SELENIUM_WAIT_DEBT, '#464')}`,
Expand Down
7 changes: 0 additions & 7 deletions packages/conformance/src/gap-matchers.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { ADAPTERS } from './adapters.js'
import { SCENARIO_TESTS } from './constants.js'
import { testLabel } from './step-alignment.js'
import type { AdapterId, MessageMatcher } from './types.js'

Expand Down Expand Up @@ -49,12 +48,6 @@ export function moved(
}
}

/** A scenario test the suite tree reports in a given state. */
export function reportedAs(test: number, state: string): MessageMatcher {
const lead = `"${SCENARIO_TESTS[test].title}" is reported ${state};`
return (message) => message.startsWith(lead)
}

export const anyOf =
(...matchers: MessageMatcher[]): MessageMatcher =>
(message) =>
Expand Down
7 changes: 5 additions & 2 deletions packages/nightwatch-devtools/src/helpers/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,13 @@ export {
export function determineTestState(
testcase: NightwatchTestCase
): 'passed' | 'failed' | 'skipped' {
if (testcase.passed === 0 && testcase.failed === 0) {
if (testcase.failed > 0 || testcase.errors > 0) {
return 'failed'
}
if (testcase.passed === 0 && testcase.skipped > 0) {
return 'skipped'
}
return testcase.passed > 0 && testcase.failed === 0 ? 'passed' : 'failed'
return 'passed'
}

export function getTestIcon(state: TestStats['state']): string {
Expand Down
33 changes: 33 additions & 0 deletions packages/nightwatch-devtools/tests/utils.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,17 @@ import {

describe('determineTestState', () => {
it.each([
[
{
passed: 0,
failed: 0,
errors: 0,
skipped: 0,
time: '0',
assertions: []
},
'passed'
],
[
{
passed: 0,
Expand Down Expand Up @@ -54,6 +65,17 @@ describe('determineTestState', () => {
},
'failed'
],
[
{
passed: 0,
failed: 0,
errors: 1,
skipped: 0,
time: '0',
assertions: []
},
'failed'
],
[
{
passed: 1,
Expand All @@ -64,6 +86,17 @@ describe('determineTestState', () => {
assertions: []
},
'failed'
],
[
{
passed: 1,
failed: 0,
errors: 1,
skipped: 0,
time: '0',
assertions: []
},
'failed'
]
])('maps testcase to %j → %s', (tc, expected) => {
expect(determineTestState(tc as never)).toBe(expected)
Expand Down
Loading