Add a support to report failures which happen session/test setup/tierdown #65

Activity

1. gregweb commented on Sep 10, 2026

   

   gregweb

   on Sep 10, 2026

   Last edited by gregweb

   More actions

   May you please elaborate a little my questions and comments?

   To your first item:

   • IMHO if a framework does not report the tests not being processed or run in some way (e.g. by listing them under results.tests[].status as skipped or pending), I don't see any way to work around this issue without additional framework external tooling. Or did I understood you wrongly?
   • How is this related to "shared test/testsuite"?
   • How is this related to "report failures ..." in this issues title?

   Regarding the second item: To my understanding of the CTRF spec, ...

   • a failure in the tests tear down may (or should) be reported as failed by results.tests[].status. There is no distinction between failures in the fixture (set up/tear down) reported as "error" and failures in the test code reported as "failure" as being typical for xUnit frameworks. For this results.tests[].rawStatus could be used.
   • test suites are not modeled by CTRF and probably won't be in future. So you'll end in inventing your own extra objects to somehow document the affected suite. Maybe under results.environment.extra.<your-namespace>.suitesWithErrors?
   • a failure in the test run (mentioned here for completeness sake) could be documented by setting results.environment.healthyto false and maybe additional information results.environment.extra.<your-namespace>.runErrors?

   Regarding "smuggling the results behind the dummy test failed test case" and "like testFrameworkChecks":

   • I'm not sure if I understand this correctly: May you please sketch a small CTRF JSON example to give some extra information?
2. Ma11hewThomas commented on Oct 25, 2026

   

   Ma11hewThomas

   on Oct 25, 2026

   Collaborator

   More actions

   Thank you both for the input and discussion.

   We plan to add optional results.errors support in CTRF 0.3.0 for global errors associated with the run rather than an individual test result, including shared setup and teardown failures:

   {
     "results": {
       "errors": [
         {
           "message": "Shared teardown failed",
           "trace": "Error: Database cleanup failed\n    at cleanup.js:24:5"
         }
       ]
     }
   }

   Each entry can contain optional message, trace and extra fields.

   This gives these errors a standard home without creating dummy test results or changing the test counts. Global errors can be reported even when every reported test passed. Failures attributed to an individual test remain on that test result.

   We’re keeping the initial shape simple. It does not yet identify the affected suite or tests, distinguish lifecycle phases, or define an overall run status. It also cannot recover tests that the framework never reported, so that part of the question remains open.

   Thank you again for helping shape CTRF.
3. jamesarosen commented on Oct 31, 2026

   

   jamesarosen

   on Oct 31, 2026

   More actions

   Thank you for working on this, @Ma11hewThomas. I opened ctrf-io/playwright-ctrf-json-reporter#123 before I'd seen this thread. I agree run-level errors shouldn't become synthetic tests if we can avoid it. I surveyed how reporters in other ecosystems handle this. The full write-up is below. The research was AI-assisted. Each claim links to pinned source or docs.

   Short version: yes to results.errors[].{message, trace, extra} as drafted in #88. It's the dominant pattern wherever a format has somewhere to put run-level errors: xUnit <errors>, Allure Globals.errors, TRX RunInfos, Playwright JSON errors, and RSpec messages. xUnit v3 already ships a CTRF writer that has to park these errors under extra because the field doesn't exist yet.

   A few "yes, and" suggestions:

   1. And make a non-empty list a dependable signal. Suggested wording: a producer SHOULD emit at least one results.errors entry when the framework reports the run as unsuccessful but no test result failed (for example, Playwright's FullResult.status of failed, timedout, or interrupted). Some runs fail with only a status and no error object. Without this rule, those runs produce a report that looks green, and the "Consumers MUST account for reported global errors" text in Add global errors to results #88 has nothing to act on.
   2. And define merge semantics. Suggested wording: when merging documents (shards), results.errors SHOULD be concatenated, not de-duplicated, and producers MAY record the shard in extra. Playwright's merge-reports and Allure 3 both concatenate. Today ctrf merge and ctrf-js merge() keep only the first document's extra and environment, so errors from later shards are silently dropped.
   3. And, as a later follow-up, an optional run outcome. Every format that keeps errors separate from tests and still wants the report to carry a verdict adds one: TRX ResultSummary/@outcome, Cucumber TestRunFinished.success, Jest success, node:test test:summary.success, and Datadog session status. I don't think this should block 0.3.0, but it's the natural next step for the "overall-run-outcome questions" that Add global errors to results #88 leaves open.
   4. And plan consumer adoption alongside the release. None of the CI consumers I checked read run-level error fields. That includes ctrf-io/github-test-reporter, which gates exit-on-fail only on summary.failed. Having exit-on-fail also fail on a non-empty results.errors would put the MUST into practice for the most common CTRF consumer.

   Once this lands, I'm happy to rework playwright-ctrf-json-reporter#123 to emit results.errors this way.

   Report: how test reporters represent run-level failures

   Context

   • playwright-ctrf-json-reporter#123 adds a synthetic failed test when Playwright's run status (FullResult.status) isn't passed and no test failed.
   • The maintainer prefers to align with this issue.
   • That alignment is now concrete: ctrf#88 is open and adds results.errors: [{message, trace, extra}]. Its diff says "Global errors MUST NOT be counted as test results" and "Consumers MUST account for reported global errors when interpreting the run". The PR body says "overall-run-outcome questions remain open."

   So playwright-ctrf-json-reporter#123 as written would contradict the upcoming spec text.

   Finding 1: frameworks split into three patterns, decided mostly by output format

   Pattern A: a synthetic test entry. Almost all of these write JUnit XML, which has no top-level place for errors.

   • jest-junit adds "Test execution failure: could be caused by test hooks like 'afterAll'". Its handling of suite load errors is off by default and enabled with reportTestSuiteErrors.
   • Vitest JUnit gained a synthetic vitest unhandled errors suite in May 2026 (PR #10244). It counts these as errors, not failures.
   • go-junit-report explains why: "JUnit doesn't have a good way of dealing with build or runtime errors… so we create a single failing test." gotestsum emits a synthetic TestMain case.
   • Maven Surefire and Gradle synthesize initializationError / executionError tests.
   • Others that do the same: Python unittest's _ErrorHolder ("setUpClass (mod.Class)"), Mocha's "before all" hook runnables, Cypress's "An uncaught error was detected outside of a test", Kotest placeholders, and MSTest's [Class ClassCleanup] result.

   Pattern B: a separate list of run-level errors, with test counts unchanged. This is what formats with a native place for errors do.

   • Playwright JSON has top-level errors, and the HTML and blob reporters keep their own lists too.
   • xUnit v2 XML has <errors>, which it describes as "separate from failing tests (for example, errors that occurred while cleaning up…)".
   • VSTest TRX has RunInfos.
   • Allure has Globals.errors.
   • RSpec has errors_outside_of_examples_count plus messages.
   • Jest uses testExecError on the file result plus numRuntimeErrorTestSuites. Its source gives the reason for afterAll: "Attaching afterAll errors to each test makes execution flow too complicated, so we'll consider them to be global."
   • The closest precedent: xUnit v3 ships its own CTRF writer. It puts cleanup and fatal errors in results.extra.suites[].errors[] and counts them only in summary.other. tests and failed stay untouched.

   Pattern C: the error is attached to a real test or to its container.

   • pytest attaches a session-fixture teardown error to the last test and reports it as error, not failed.
   • On before(:context) / @BeforeAll / OneTimeSetUp failures, RSpec, Jest, NUnit and JUnit's legacy XML mark every affected test failed. JUnit Platform and Open Test Reporting give the container itself a FAILED status.

   Dropping the error is the bug the ecosystem keeps fixing.

   • Playwright's own JUnit reporter has no onError handler, so it reports exit 1 with 0 failures.
   • allure-playwright has onError(): void {}.
   • Vitest JSON, node:test JUnit and cargo2junit also drop it.
   • Vitest's JUnit reporter fixed exactly this in 2026.

   Finding 2: an error list is almost always paired with a run outcome

   When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:

   • Jest: success is false when numRuntimeErrorTestSuites > 0.
   • TRX: ResultSummary/@outcome="Failed" even when every test passed, with the run-level counters left alone.
   • Cucumber Messages: TestRunFinished.success is required, along with an exception field.
   • node:test: test:summary.success is set false for uncaught errors outside tests, without changing failed (harness.js).
   • pytest-json-report: top-level exitcode.
   • Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.

   The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.

   Finding 3: consumers ignore run-level error fields

   • ctrf-io/github-test-reporter gates only on summary.failed > 0. Nothing in its src reads extra.errors or environment.healthy.
   • mikepenz/action-junit-report ignores suite-level errors. The maintainer confirmed it in issue #1005, where a report with 11 suite-level errors and 0 failures passed.
   • dorny/test-reporter computes its verdict from test cases only.
   • GitLab doesn't parse the suite-level errors attribute, and says job status comes from the script's exit code.
   • The one consumer that converts a suite-level error is Jenkins: a suite-level <error> becomes a synthetic <init> test (SuiteResult.java).

   So the #88 sentence "Consumers MUST account for reported global errors" is ahead of today's tooling.

   Finding 4: merging is where run-level errors get lost

   • Playwright's merge-reports replays every shard's onError with no de-duplication. It merges run status worst-first (failed, then timedout, then interrupted, then passed). Allure 3 also appends errors without de-duplication.
   • ctrf merge in ctrf-cli keeps extra and environment from the first shard only, so errors from shards 2 through N are silently lost.
   • The ctrf-js merge() keeps the first value for each key and rebuilds the result without any results.errors handling.
   • Add global errors to results #88 says nothing about merging.

   Options

   Option	Precedent	Changes counts?	Gateable today?
   A. Synthetic failed test (playwright-ctrf-json-reporter#123 as written)	JUnit-world reporters, Surefire, Gradle, unittest, Cypress	Yes	Yes
   B. results.errors only	Playwright JSON, xUnit v2, Allure, RSpec	No	No
   C. B, plus always emit an error entry when the run fails	Jest's testExecError, xUnit v3 CTRF writer	No	Only for consumers that read errors
   D. C, plus a spec-level run outcome	TRX, Cucumber, Jest, node:test, Datadog	No	Yes, once consumers adopt it
   E. Opt-in flag that also adds a synthetic test	jest-junit's reportTestSuiteErrors	Only when enabled	Yes, when enabled
4. Ma11hewThomas commented on Oct 31, 2026

   

   Ma11hewThomas

   on Oct 31, 2026

   Last edited by Ma11hewThomas

   Collaborator

   More actions

   Thank you for the research @jamesarosen

   At a high level, I like these ideas for the spec.

   Could you open a separate issue in the this repo⁠ for each:

   making a non-empty error list a dependable signal,

   defining error merge semantics,

   and an optional run outcome?

   Thanks again!

   👍React with 👍1Reacted by James A Rosen