Skip to content

Add optional threadId to attempt history entries #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alexshamrai wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add optional threadId to attempt history entries #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- Optional `threadId` on attempt history entries, using the same contract as Test
`threadId`, so previous attempts can record the thread or worker that executed
them (#74).

### Changed

- Clarified that Test `threadId` identifies the thread or worker that executed the
final attempt, and recommended qualifying `threadId` values when test cases run
across processes or hosts (#74).

## [0.2.0] - 2026-10-10

### Added
Expand Down
1 change: 1 addition & 0 deletions examples/comprehensive.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@
"duration": 5100,
"start": 1609459203500,
"stop": 1609459208600,
"threadId": "worker-1",
"message": "Payment gateway timeout",
"trace": "Error: Payment gateway timeout\n at processPayment (payment.js:23:10)",
"attachments": [
Expand Down
6 changes: 5 additions & 1 deletion schema/ctrf.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,10 @@
"description": "Attempt end time (milliseconds since Unix epoch)",
"type": "integer"
},
"threadId": {
"description": "Thread or worker that executed this attempt",
"type": "string"
},
"attachments": {
"description": "Artifacts from this attempt",
"type": "array",
Expand Down Expand Up @@ -397,7 +401,7 @@
}
},
"threadId": {
"description": "Thread or worker identifier",
"description": "Thread or worker that executed the final attempt represented by the test object",
"type": "string"
},
"browser": {
Expand Down
25 changes: 21 additions & 4 deletions spec/ctrf.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1147,11 +1147,12 @@ If present, it MUST be an array of strings.
### 9.25. `threadId`

**Description:**
Identifies the thread or worker on which the test executed.
Identifies the thread or worker that executed the final attempt represented by this test object. Earlier attempts can record their own `threadId` in `retryAttempts` (Section 11).

**Requirements:**
`threadId` is OPTIONAL.
If present, it MUST be a string.
If present, it MUST be a string.
When test cases run in more than one process or on more than one host, producers SHOULD qualify the value with the process or host to avoid collisions, for example, `"host-a/worker-1"`.

### 9.26. `browser`

Expand Down Expand Up @@ -1549,6 +1550,16 @@ The timestamp indicating when the attempt ended, in milliseconds since the Unix
If present, it MUST be an integer.
If both `start` and `stop` are present, `stop` SHOULD be greater than or equal to `start`.

### 11.12.1. `threadId`

**Description:**
Identifies the thread or worker that executed this attempt.

**Requirements:**
`threadId` is OPTIONAL.
If present, it MUST be a string.
When test cases run in more than one process or on more than one host, producers SHOULD qualify the value with the process or host to avoid collisions, for example, `"host-a/worker-1"`.

### 11.13. `attachments`

**Description:**
Expand Down Expand Up @@ -2427,6 +2438,10 @@ to this specification.
"description": "Attempt end time (milliseconds since Unix epoch)",
"type": "integer"
},
"threadId": {
"description": "Thread or worker that executed this attempt",
"type": "string"
},
"attachments": {
"description": "Artifacts from this attempt",
"type": "array",
Expand Down Expand Up @@ -2487,7 +2502,7 @@ to this specification.
}
},
"threadId": {
"description": "Thread or worker identifier",
"description": "Thread or worker that executed the final attempt represented by the test object",
"type": "string"
},
"browser": {
Expand Down Expand Up @@ -3337,10 +3352,12 @@ It includes:
"duration": 1500,
"message": "Timeout waiting for response",
"stdout": ["Submitting login form"],
"stderr": ["Request timed out"]
"stderr": ["Request timed out"],
"threadId": "worker-1"
}
],
"stdout": ["Test completed successfully"],
"threadId": "worker-2",
"insights": {
"passRate": {
"current": 0.9,
Expand Down
190 changes: 190 additions & 0 deletions tests/normative/type-validation.test.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1234,6 +1234,196 @@
]
}
}
},
{
"description": "RetryAttempt with threadId as number instead of string",
"valid": false,
"data": {
"reportFormat": "CTRF",
"specVersion": "0.2.0",
"results": {
"tool": {
"name": "runner"
},
"summary": {
"tests": 1,
"passed": 1,
"failed": 0,
"pending": 0,
"skipped": 0,
"other": 0,
"start": 1609459200000,
"stop": 1609459201000
},
"tests": [
{
"name": "test",
"status": "passed",
"duration": 100,
"retries": 1,
"retryAttempts": [
{
"attempt": 1,
"status": "failed",
"threadId": 12345
}
]
}
]
}
}
},
{
"description": "RetryAttempt with threadId as null instead of string",
"valid": false,
"data": {
"reportFormat": "CTRF",
"specVersion": "0.2.0",
"results": {
"tool": {
"name": "runner"
},
"summary": {
"tests": 1,
"passed": 1,
"failed": 0,
"pending": 0,
"skipped": 0,
"other": 0,
"start": 1609459200000,
"stop": 1609459201000
},
"tests": [
{
"name": "test",
"status": "passed",
"duration": 100,
"retries": 1,
"retryAttempts": [
{
"attempt": 1,
"status": "failed",
"threadId": null
}
]
}
]
}
}
},
{
"description": "RetryAttempt with threadId as boolean instead of string",
"valid": false,
"data": {
"reportFormat": "CTRF",
"specVersion": "0.2.0",
"results": {
"tool": {
"name": "runner"
},
"summary": {
"tests": 1,
"passed": 1,
"failed": 0,
"pending": 0,
"skipped": 0,
"other": 0,
"start": 1609459200000,
"stop": 1609459201000
},
"tests": [
{
"name": "test",
"status": "passed",
"duration": 100,
"retries": 1,
"retryAttempts": [
{
"attempt": 1,
"status": "failed",
"threadId": true
}
]
}
]
}
}
},
{
"description": "RetryAttempt with threadId as array instead of string",
"valid": false,
"data": {
"reportFormat": "CTRF",
"specVersion": "0.2.0",
"results": {
"tool": {
"name": "runner"
},
"summary": {
"tests": 1,
"passed": 1,
"failed": 0,
"pending": 0,
"skipped": 0,
"other": 0,
"start": 1609459200000,
"stop": 1609459201000
},
"tests": [
{
"name": "test",
"status": "passed",
"duration": 100,
"retries": 1,
"retryAttempts": [
{
"attempt": 1,
"status": "failed",
"threadId": []
}
]
}
]
}
}
},
{
"description": "RetryAttempt with threadId as object instead of string",
"valid": false,
"data": {
"reportFormat": "CTRF",
"specVersion": "0.2.0",
"results": {
"tool": {
"name": "runner"
},
"summary": {
"tests": 1,
"passed": 1,
"failed": 0,
"pending": 0,
"skipped": 0,
"other": 0,
"start": 1609459200000,
"stop": 1609459201000
},
"tests": [
{
"name": "test",
"status": "passed",
"duration": 100,
"retries": 1,
"retryAttempts": [
{
"attempt": 1,
"status": "failed",
"threadId": {}
}
]
}
]
}
}
}
]
}
46 changes: 46 additions & 0 deletions tests/normative/valid-documents.test.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,52 @@
]
}
}
},
{
"description": "Previous attempt threadId differs from the final attempt threadId",
"valid": true,
"data": {
"reportFormat": "CTRF",
"specVersion": "0.2.0",
"results": {
"tool": {
"name": "runner"
},
"summary": {
"tests": 1,
"passed": 1,
"failed": 0,
"pending": 0,
"skipped": 0,
"other": 0,
"flaky": 1,
"start": 1609459200000,
"stop": 1609459201000
},
"tests": [
{
"name": "checkout accepts a saved card",
"status": "passed",
"duration": 500,
"start": 1609459200500,
"stop": 1609459201000,
"threadId": "fork-2/Test worker",
"flaky": true,
"retries": 1,
"retryAttempts": [
{
"attempt": 1,
"status": "failed",
"duration": 400,
"start": 1609459200000,
"stop": 1609459200400,
"threadId": "fork-1/Test worker"
}
]
}
]
}
}
}
]
}
Loading