Skip to content

Replace the invalid 2.13.1 reused cancellation-clock evidence with valid redrive successors #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rmcdaniel merged 3 commits into from
Nov 3, 2026
Merged

Replace the invalid 2.13.1 reused cancellation-clock evidence with valid redrive successors #238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bd41be6
Replace the invalid 2.13.1 reused cancellation-clock evidence
rmcdaniel Nov 3, 2026
52940fb
Bind the successor cancellation exactly and parse times on Python 3.10
rmcdaniel Nov 3, 2026
262eb6e
Bind the copied timer schedule to the source run and plain timers
rmcdaniel Nov 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- Replay now enforces the recorded search attributes (Workflow protocol 1.20 replay identity). A workflow whose `upsert_search_attributes` differs from its `SearchAttributesUpserted` history fails replay with a `NonDeterministicReplayError` whose detail starts `search_attribute_value_mismatch`, as memo replay already does for memos. Values are compared as Core's `UpsertSearchAttributesCall` normalizes them (trimmed strings and list items, a blank string as null) and with their types (`1`, `1.0`, `True` and `"1"` differ). An authored tuple is checked as the JSON list it is sent as. Values outside that portable domain, on either side, are refused rather than coerced: non-string or non-portable keys, objects, sets, bytes, maps, non-string list items, non-finite floats and out-of-range integers. A `SearchAttributesUpserted` event without `attributes` is refused rather than skipped, with a detail starting `search_attribute_update_missing`. One without a workflow sequence is refused with `durable_command_sequence_missing` (or `durable_command_sequence_invalid` for zero, negative, string, float or bool) rather than given one, and a second upsert at the same sequence with `duplicate_search_attribute_update`. An upsert that shares its workflow sequence with any other durable command record is refused with `durable_command_sequence_collision`. Every lifecycle row that carries a command's sequence claims it, including activity heartbeat, retry and cancellation-acknowledged rows, child cancellation rows, service operation scheduling, cancellation scope openings, continue-as-new, and cooperative and scope cancellation deliveries, which claim every sequence in their `sequence_span` (absent or null means 1, as in Core; any other value that is not a positive integer is refused with `durable_command_sequence_invalid`); events that only point at a step, such as a signal receipt, an update or a cancellation request, claim no sequence. When the history records `attribute_types`, it must type exactly the update's non-null keys, and each type must be a known type that can hold the value; anything else, including a malformed type value, fails with `search_attribute_type_mismatch`. The SDK's upsert declares no types and the Server records a key's namespace-registered type ahead of the inferred one, so a string may be recorded as `keyword`, `string` or `datetime` and an integer as `int` or `float`; search-attribute type identity is therefore not pinned for Python. History without `attribute_types` keeps value-only identity. Replay diagnostics name reasons, keys, JSON types, sequences and event types, never a recorded or authored value. (#235)

### Corrected

- **2.13.1 claim withdrawn (#237).** The 2.13.1 entry said "The cancellation clock prefers a reused step's `reused_recorded_at`." As an observable result, that is incorrect. The test behind it was not a history Workflow 2.19 can write: it had no redrive `WorkflowStarted` and delivered the cancellation before the reused step. Redrive writes the successor's start and its reused prefix together, so a successor cancellation always follows the prefix and `remaining()` is computed from that delivery. A reused event's `reused_recorded_at` therefore cannot reach the budget, and the preference changes no observable result. Replay behavior is unchanged. The 2.13.1 start-clock change (`ctx.now()` from `replayed_started_at`) is unaffected. Valid successor fixtures replace the evidence, and the 2.13.1 fixture `redrive-start-replayed-clock`, which has only a partial redrive envelope, is registered as non-qualifying.

### Added

- Replay of Workflow 2.19 redrive reuse is qualified for `MemoUpserted` and for `SearchAttributesUpserted` value identity. Neither event is a replay clock source: `ctx.now()` stays the persisted start time, which on a redriven run is the failed run's start, and the cancellation budget does not move on an upsert. Search-attribute type identity is not qualified, because the Server records namespace-registered types the SDK cannot see. Signal waits (`SignalWaitOpened`, `SignalReceived`, `SignalApplied`) are not qualified for Python: the SDK has no signal-wait primitive. Replaying such reused events through a signal handler applies the signal once and takes no time from them, but that does not qualify signal waits. (#235)
Expand Down
162 changes: 162 additions & 0 deletions tests/fixtures/replay_regressions/redrive-successor-reused-cancellation-clock.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
{
"$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json",
"fixture_schema": "durable-workflow.replay-regression/v1",
"id": "redrive-successor-reused-cancellation-clock",
"protocol_version": "1.20",
"bindings": [
"python"
],
"workflow": {
"type": "tests.replay.redrive-successor-cancellation-clock",
"input": [
"Taylor",
5
],
"payload_codec": "avro",
"workflow_id": "regression-corpus-redrive-successor-cancellation",
"run_id": "01JBREDRIVESUCCESSORRUN0000"
},
"history": [
{
"sequence": 1,
"event_type": "WorkflowStarted",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"workflow_type": "tests.replay.redrive-successor-cancellation-clock",
"workflow_instance_id": "regression-corpus-redrive-successor-cancellation",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"workflow_command_id": "01JBREDRIVECOMMAND00000000",
"continued_from_run_id": "01JBREDRIVESOURCERUN000000",
"resume_step_sequence": 3,
"recovery_kind": "redrive",
"execution_deadline_at": null,
"run_deadline_at": null,
"replayed_started_at": "2026-11-03T00:00:00+00:00",
"workflow_link_id": "01JBREDRIVELINK00000000000"
}
},
{
"sequence": 2,
"event_type": "ActivityCompleted",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"activity_execution_id": "01JBREDRIVESUCCESSORACT000",
"activity_type": "tests.redrive.first",
"sequence": 1,
"result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=",
"payload_codec": "avro",
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_activity_execution_id": "01JBREDRIVESOURCEACTIVITY0",
"reused_recorded_at": "2026-11-03T00:00:00+00:00"
}
},
{
"sequence": 3,
"event_type": "TimerScheduled",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"timer_id": "01JBREDRIVESUCCESSORTIMER0",
"sequence": 2,
"delay_seconds": 5,
"fire_at": "2026-11-03T00:00:05.000000Z"
}
},
{
"sequence": 4,
"event_type": "TimerFired",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"timer_id": "01JBREDRIVESUCCESSORTIMER0",
"sequence": 2,
"delay_seconds": 5,
"fire_at": "2026-11-03T00:00:05.000000Z",
"fired_at": "2026-11-03T00:00:06.000000Z",
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_recorded_at": "2026-11-03T00:00:06+00:00"
}
},
{
"sequence": 5,
"event_type": "ActivityScheduled",
"recorded_at": "2026-11-03T00:05:00.500000+00:00",
"payload": {
"activity_execution_id": "01JBREDRIVESUCCESSORACT001",
"activity_type": "tests.redrive.second",
"sequence": 3
}
},
{
"sequence": 6,
"event_type": "CooperativeCancellationRequested",
"recorded_at": "2026-11-03T00:05:01+00:00",
"payload": {
"workflow_command_id": "successor-request",
"workflow_instance_id": "regression-corpus-redrive-successor-cancellation",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"command_type": "request_cancellation",
"cleanup_deadline_at": "2026-11-03T00:05:31.000000Z",
"cancellation": {
"schema": "durable-workflow.cancellation-context/v1",
"request_id": "successor-request",
"root_request_id": "successor-request",
"root_workflow_instance_id": "regression-corpus-redrive-successor-cancellation",
"root_workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"parent_request_id": null,
"reason": null,
"requester": {
"type": "operator",
"id": "fixture-operator"
},
"source": "control_plane",
"requested_at": "2026-11-03T00:05:01.000000Z",
"cleanup_deadline_at": "2026-11-03T00:05:31.000000Z",
"lineage": [
{
"request_id": "successor-request",
"workflow_instance_id": "regression-corpus-redrive-successor-cancellation",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000"
}
]
}
}
},
{
"sequence": 7,
"event_type": "CooperativeCancellationDelivered",
"recorded_at": "2026-11-03T00:05:02+00:00",
"payload": {
"workflow_command_id": "successor-request",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"sequence": 3,
"call_kind": "activity",
"cancellation": {
"schema": "durable-workflow.cancellation-context/v1",
"request_id": "successor-request",
"root_request_id": "successor-request",
"root_workflow_instance_id": "regression-corpus-redrive-successor-cancellation",
"root_workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"parent_request_id": null,
"reason": null,
"requester": {
"type": "operator",
"id": "fixture-operator"
},
"source": "control_plane",
"requested_at": "2026-11-03T00:05:01.000000Z",
"cleanup_deadline_at": "2026-11-03T00:05:31.000000Z",
"lineage": [
{
"request_id": "successor-request",
"workflow_instance_id": "regression-corpus-redrive-successor-cancellation",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000"
}
]
}
}
}
],
"expected": {
"type": "complete_workflow",
"result": 29.0
}
}
90 changes: 90 additions & 0 deletions tests/fixtures/replay_regressions/redrive-successor-reused-markers.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{
"$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json",
"fixture_schema": "durable-workflow.replay-regression/v1",
"id": "redrive-successor-reused-markers",
"protocol_version": "1.20",
"bindings": [
"python"
],
"workflow": {
"type": "tests.replay.redrive-successor-markers",
"input": [
"Taylor"
],
"payload_codec": "avro",
"workflow_id": "regression-corpus-redrive-successor-markers",
"run_id": "01JBREDRIVESUCCESSORRUN0000"
},
"history": [
{
"sequence": 1,
"event_type": "WorkflowStarted",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"workflow_type": "tests.replay.redrive-successor-markers",
"workflow_instance_id": "regression-corpus-redrive-successor-markers",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"workflow_command_id": "01JBREDRIVECOMMAND00000000",
"continued_from_run_id": "01JBREDRIVESOURCERUN000000",
"resume_step_sequence": 4,
"recovery_kind": "redrive",
"execution_deadline_at": null,
"run_deadline_at": null,
"replayed_started_at": "2026-11-03T00:00:00+00:00",
"workflow_link_id": "01JBREDRIVELINK00000000000"
}
},
{
"sequence": 2,
"event_type": "SideEffectRecorded",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"sequence": 1,
"result": "wwHioz3/VYAiNwogMDEyMzQ1Njc4OWFiY2RlZg==",
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_recorded_at": "2026-11-03T00:00:00+00:00"
}
},
{
"sequence": 3,
"event_type": "VersionMarkerRecorded",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"sequence": 2,
"change_id": "redrive-markers",
"version": 1,
"min_supported": -1,
"max_supported": 1,
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_recorded_at": "2026-11-03T00:00:00+00:00"
}
},
{
"sequence": 4,
"event_type": "ActivityCompleted",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"activity_execution_id": "01JBREDRIVESUCCESSORACT000",
"activity_type": "tests.redrive.first",
"sequence": 3,
"result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=",
"payload_codec": "avro",
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_activity_execution_id": "01JBREDRIVESOURCEACTIVITY0",
"reused_recorded_at": "2026-11-03T00:00:01+00:00"
}
}
],
"expected": {
"command_sequence": [
{
"type": "schedule_activity",
"activity_type": "tests.redrive.second",
"arguments": [
"Hello, Taylor!",
"2026-11-03T00:00:00+00:00"
]
}
]
}
}
90 changes: 90 additions & 0 deletions tests/fixtures/replay_regressions/redrive-successor-reused-timer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{
"$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json",
"fixture_schema": "durable-workflow.replay-regression/v1",
"id": "redrive-successor-reused-timer",
"protocol_version": "1.20",
"bindings": [
"python"
],
"workflow": {
"type": "tests.replay.redrive-successor-timer",
"input": [
"Taylor",
5
],
"payload_codec": "avro",
"workflow_id": "regression-corpus-redrive-successor-timer",
"run_id": "01JBREDRIVESUCCESSORRUN0000"
},
"history": [
{
"sequence": 1,
"event_type": "WorkflowStarted",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"workflow_type": "tests.replay.redrive-successor-timer",
"workflow_instance_id": "regression-corpus-redrive-successor-timer",
"workflow_run_id": "01JBREDRIVESUCCESSORRUN0000",
"workflow_command_id": "01JBREDRIVECOMMAND00000000",
"continued_from_run_id": "01JBREDRIVESOURCERUN000000",
"resume_step_sequence": 3,
"recovery_kind": "redrive",
"execution_deadline_at": null,
"run_deadline_at": null,
"replayed_started_at": "2026-11-03T00:00:00+00:00",
"workflow_link_id": "01JBREDRIVELINK00000000000"
}
},
{
"sequence": 2,
"event_type": "ActivityCompleted",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"activity_execution_id": "01JBREDRIVESUCCESSORACT000",
"activity_type": "tests.redrive.first",
"sequence": 1,
"result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=",
"payload_codec": "avro",
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_activity_execution_id": "01JBREDRIVESOURCEACTIVITY0",
"reused_recorded_at": "2026-11-03T00:00:00+00:00"
}
},
{
"sequence": 3,
"event_type": "TimerScheduled",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"timer_id": "01JBREDRIVESUCCESSORTIMER0",
"sequence": 2,
"delay_seconds": 5,
"fire_at": "2026-11-03T00:00:05.000000Z"
}
},
{
"sequence": 4,
"event_type": "TimerFired",
"recorded_at": "2026-11-03T00:05:00+00:00",
"payload": {
"timer_id": "01JBREDRIVESUCCESSORTIMER0",
"sequence": 2,
"delay_seconds": 5,
"fire_at": "2026-11-03T00:00:05.000000Z",
"fired_at": "2026-11-03T00:00:06.000000Z",
"reused_from_run_id": "01JBREDRIVESOURCERUN000000",
"reused_recorded_at": "2026-11-03T00:00:06+00:00"
}
}
],
"expected": {
"command_sequence": [
{
"type": "schedule_activity",
"activity_type": "tests.redrive.second",
"arguments": [
"Hello, Taylor!"
]
}
]
}
}
Loading
Loading