[rmcdaniel]

Fixes #237. Head 262eb6ef addresses the interim review of 52940fb7 (F1: copied timer schedule bound to the source run and to plain timers), on top of Security's hold on bd41be60 (comment 6617712901: Python 3.10 time parsing and exact successor-cancellation binding). This changes tests, fixtures and the CHANGELOG only. There is no product change, the corpus policy and scripts/ci are unchanged, and every existing fixture is byte-identical to main (7dbeca44).

What was invalid

• test_a_reused_step_keeps_its_original_time_for_the_remaining_budget (tests/test_cancellation_remaining_time.py, Replay a redriven run from the failed run's start time #233, shipped in 2.13.1). Its history has:

  • no redrive WorkflowStarted;
  • the cancellation delivered at 00:08, before a "reused" step;
  • a reused_recorded_at (00:12) later than the delivery.

  Its 18.123456 depends on that order; the valid order gives 22.123456. It is the only evidence for the 2.13.1 clause "The cancellation clock prefers a reused step's reused_recorded_at".

• redrive-start-replayed-clock.json (2.13.1) has a partial envelope: only recovery_kind and replayed_started_at, with no history sequence, identity, link or resume step. It has no reused steps and asserts only ctx.now(), which is correct.

  • Decision: ambiguous, not a clock defect. It is registered as non-qualifying, and a valid successor re-establishes its start-clock claim.

What replaces it

The unit test is removed, not rewritten. It is not a corpus fixture, and no valid history can observe the preference it asserted, so there is nothing true to rewrite it into. A comment in its place points to the new evidence.

Three successor fixtures (protocol 1.20) are built to Workflow 28893f97 FailedRunRedrivePlan::successorEvents() (lines 406-485) and the redrive WorkflowStarted from DefaultWorkflowControlPlane::redrive(). They mirror sdk-php#249's fixtures, with Python workflows and activity types. Each has the full envelope, a Core-identical reused prefix and resume_step_sequence = highest reused step + 1:

• redrive-successor-reused-markers
  • Reuses side effect 1, version marker 2 and activity 3; resume is 4.
  • The side-effect callable runs 0 times, and the markers replay as "0123456789abcdef" and version 1.
  • The next command is tests.redrive.second(["Hello, Taylor!", "2026-11-03T00:00:00+00:00"]). The second argument is ctx.now(), i.e. replayed_started_at. This is the replacement for redrive-start-replayed-clock.
• redrive-successor-reused-timer
  • Reuses activity 1 and fired timer 2; resume is 3.
  • The timer is Core's copy: a TimerScheduled with a new timer_id and no reuse keys, followed by the reused TimerFired. No timer is started again.
• redrive-successor-reused-cancellation-clock
  • The same successor, then its own ActivityScheduled step 3 at 00:05:00.5.
  • Its own cancellation is requested at 00:05:01 (deadline 00:05:31) and delivered to step 3 at 00:05:02. The request and delivery carry the instance, run, command_type and cancellation context, as Core writes them.
  • remaining() = 29.0.

Corrected claim: non-leakage

In a valid history the reused prefix comes before the successor's delivery, and Python's cancellation clock starts at the delivery. Reused times therefore cannot reach remaining().

This is not fail-before/pass-after evidence for preferring reused_recorded_at: no valid history can observe that preference. Product code is unchanged; _observe_cancellation_replay_time still prefers it, which is harmless.

Tests in tests/test_redrive_successor_evidence.py (118 cases):

• Valid retiming. Every reused event's reused_recorded_at is moved to 00:00:00 and to 00:04:59.999999, on all three successors. Each history still passes the check, and the commands are unchanged.

• Leaking-clock probe. The reused ActivityCompleted, and separately the reused TimerFired, is retimed to 00:05:20 (recorded_at and reused_recorded_at), after the 00:05:02 delivery.

  • A probe clock that reads every event up to the delivery reports 11.0 on that history and 29.0 on the original. The test asserts both.
  • The SDK still returns 29.0, with identical commands.
  • The retimed history fails the successor check, so it is a probe, not a history redrive writes.

• Positive control. The same 00:05:20 retiming applied to the delivery gives 11.0, so the assertion detects any event that does reach the clock.

• 2.13.1 order. Moving the successor's request and delivery ahead of the reused prefix fails the check (reused events must directly follow WorkflowStarted).

• Resume: −1, +1, null and the string form, on each new successor.

• Copied timer. On the timer and cancellation successors, each of these fails:

  • the schedule carrying reused_from_run_id, or carrying reused_recorded_at;
  • another timer_id, or a blank one;
  • another step;
  • the schedule recorded after the start;
  • the schedule moved after the fired timer;
  • no schedule, or no fired timer;
  • the schedule naming a bogus run, or the successor run;
  • the schedule carrying timer_kind, cancellation_scope_id, cancellation_cleanup, parallel_group_id, condition_wait_id or signal_wait_id.

  Fail-before: these last 8 shapes × 2 successors (16 cases) passed at 52940fb7. Controls (accepted): the schedule naming the source run, or carrying a null timer_kind or cancellation_scope_id.

• Successor cancellation. Each of these fails:

  • delivery to reused step 2 or 1;
  • the request before the start;
  • the request or the delivery on the source run;
  • the request for another instance;
  • the context rooted at the source run, or a lineage naming it;
  • a successor row carrying a reuse key;
  • the successor's ActivityScheduled on reused step 2.

• Exact cancellation binding (test_the_successor_cancellation_is_bound_exactly). 15 mutations, each on both cancellation successors (redrive-successor-reused-cancellation-clock and Enforce search-attribute replay identity and qualify redriven memo and search-attribute reuse #236's redrive-reused-upserts-cancellation-clock):

  • the context deleted from both events, from the request only, or from the delivery only;
  • a duplicate request or delivery;
  • a missing request or delivery;
  • a mismatched delivery context;
  • the delivery recorded before the request;
  • the context's request id, root request, root instance, lineage request or deadline changed;
  • delivery to unrecorded step 4.

  Fail-before: the bd41be60 check accepted all 30 cases; this one refuses all 30. Control: a delivery to step 4 after the successor recorded step 3's completion and scheduled step 4 is accepted.

• Per-field coverage. The existing parametrized envelope and per-event deletion and blank controls in tests/test_replay_clock.py now also run on the three new successors.

Validator change (check_redrive_successor, tests/test_replay_clock.py)

• Core shapes it now recognises:

  • a reused activity as one ActivityCompleted (sequence, activity_type, activity_execution_id);
  • SideEffectRecorded, VersionMarkerRecorded and TimerFired;
  • the successor's own ActivityScheduled.

• Copied timer: the one narrow allowance, the PHP rule. A TimerScheduled is accepted in the reused prefix only if it:

  • carries neither reuse key;

  • is recorded at the successor start;

  • is directly followed by a reused TimerFired with the same non-blank timer_id and the same step.

  • names no run, or the source run (continued_from_run_id), as workflow_run_id; the copy is the source run's schedule payload (FailedRunRedrivePlan.php:437-455);

  • is a plain timer, per Core's isPlainTimer (:497): no non-null timer_kind, cancellation_cleanup, cancellation_scope_id, parallel_group_id, condition_wait_id or signal_wait_id.

  Conversely, a reused TimerFired now requires that copy directly before it.

• No reused step in the successor. No successor event that occupies a step, including a cancellation delivery, may use a reused step's sequence.

• Successor cancellation, bound exactly (follows the request-identity part of sdk-rust#175's rule):

  • Shape: exactly one CooperativeCancellationRequested, followed by exactly one CooperativeCancellationDelivered.
  • Shared identity: both carry the same non-blank cancellation context, and the delivery names its request (workflow_command_id).
  • Time: the delivery is not earlier than the request.
  • Context:
    • its request_id is the request;
    • it is rooted at the successor run and instance, and its lineage ends at them, naming the request;
    • a root request (parent_request_id null) is its own root;
    • the request's cleanup_deadline_at equals the context's and does not precede the request.
  • Delivery step: the delivery targets either the first unreused step (resume_step_sequence, a call not yet scheduled; Core lets a delivery consume an unscheduled call), or a cancellable call the successor itself recorded before the delivery (ActivityScheduled, TimerScheduled or SignalWaitOpened).
  • Stricter than sdk-rust: an unrecorded later step is refused. sdk-rust allows one past the highest recorded step, but here step 3 is still pending, so step 4 cannot be reached.

• Python 3.10. The check and the evidence tests parse history times through redrive_time, which maps a trailing Z to +00:00 (Python 3.10's fromisoformat rejects Z) and requires an offset.

• Verdict sweep. I ran the merged (7dbeca44) and new checks over every corpus fixture that has a history (72). Exactly 3 verdicts change, re-run at this head: the 3 new successors go from unexpected event ActivityCompleted / unexpected event SideEffectRecorded to valid. All 69 others keep their verdict and message. Nothing that failed before passes now, except these Core-shaped successors.

Registry and gate

tests/redrive_fixture_registry.py:

Kind	Fixtures	Requirement
Valid successors	redrive-reused-upserts-clock (3), redrive-reused-signal-wait-clock (2), redrive-reused-upserts-cancellation-clock (3), redrive-reused-search-attribute-value-mismatch (3), redrive-reused-search-attribute-type-mismatch (3), redrive-reused-search-attribute-update-missing (3), and the 3 new ones (4, 3, 3)	pass the check, with that resume
Non-qualifying (2.13.1)	redrive-start-replayed-clock	byte-identical, SHA-256 70653171faf5ddba5ed264d0a7ce8bb1af97112dbd6e3bb0e39c6f155fee0fd2; fails with history order (its WorkflowStarted has no history sequence); replacement redrive-successor-reused-markers must pass
Source runs	none	—

• Gate. test_every_redrive_fixture_is_registered selects every replay fixture whose name contains redrive, that carries reused_from_run_id or reused_recorded_at, or whose WorkflowStarted has recovery_kind=redrive. That set must equal the registry, so an unregistered fixture or a stale entry fails.
• test_exactly_the_registered_successors_pass requires the set that passes the check to be exactly the registered successors.
• The non-qualifying fixture still replays through the official corpus runner, because its ctx.now() result is true.
• Left as is: tests/test_replay_clock.py::redriven_history builds in-test histories for the start-clock tests. They check only ctx.now() from replayed_started_at and make no remaining-budget claim, so they are unchanged.

CHANGELOG

A new ### Corrected section under Unreleased withdraws the 2.13.1 observable claim. The 2.13.1 section, tag and package are untouched.

Proposed text for the 2.13.1 GitHub Release (not applied)

I have not edited the release, tag or package.

• Current body: SHA-256 5d90b2f916630c6224ec319e5cf403e30a29b9177a4ef3da38795b9eafca17c0, from the raw API body. It does not end with a newline. The release is not a draft or prerelease and has no assets.
• Proposed edit: append this paragraph after the existing text with one blank line and a final newline: body + "\n\n" + paragraph + "\n", the same layout as the 4.11.1 sdk-rust edit. The full proposed body has SHA-256 db167b0e2e36fa0294aa6879e654dc0f442262bd68f0099c25c6327649ef678f.
• Wording: modelled on Security's sdk-rust#175 wording (comment 6616233097). The 2.13.1 body is the auto-generated "What's Changed" list, so the paragraph quotes the CHANGELOG clause it withdraws.

Correction (#237): the cancellation-clock test behind this release's CHANGELOG clause "The cancellation clock prefers a reused step's reused_recorded_at" was not a valid Workflow 2.19 redrive successor history: it had no redrive WorkflowStarted and delivered the cancellation before the reused step. That clause is incorrect as an observable result. In valid redrive histories, the reused prefix always precedes the successor's cancellation delivery, so remaining() is computed from that delivery and cannot observe a reused event's earlier reused_recorded_at. Preferring reused_recorded_at therefore changes no observable result and restores no cleanup budget. The start-clock change in #233 (ctx.now() and ctx.uuid7() from replayed_started_at) is unaffected. The 2.13.1 package and tag are unchanged. Valid successor evidence is in #238.

Local runs at 262eb6e

• Python 3.13 (host venv):
  • ruff check src/ tests/: clean.
  • mypy src/durable_workflow/: no issues.
  • validate-regression-corpus.py --base-ref origin/main: status: pass.
  • test-validate-regression-corpus.py: OK.
  • Full unit suite: 4,765 passed, 3 skipped, 8 failed. The same 8 fail on unmodified main: TLS fixture certificates rejected by the host's OpenSSL, and two boundary-scanner tests tied to the host git identity.
• Python 3.10.22, previous head 52940fb7. In a bounded python:3.10-slim container (general-py310, --cpus 2 --memory 2g --pids-limit 512 --init --user 501:20, removed afterwards):
  • The new evidence, clock and corpus tests: 551 passed.
  • Full unit suite: 4,745 passed, 3 skipped; 4 failed and 2 errored in test_ci_checkout.py / test_public_boundary_scanner.py. The image has no git (FileNotFoundError: 'git').

Not merged or released, and the 2.13.1 GitHub Release is unedited.

🤖 Generated with Claude Code

[rmcdaniel]

Security source hold on exact head bd41be60990791141923bcc264f9ff82c84ed902.

The withdrawal direction, immutable-fixture registry, copied-timer exception, and valid replacement shapes are sound. This head is not clear yet:

1. Python 3.10 is red on the new evidence itself. Both non-leakage cases call datetime.fromisoformat() on the Core timestamp 2026-11-03T00:05:31.000000Z; Python 3.10 rejects the trailing Z. Exact job 126262559310 failed 2 tests. Parse this through the SDK's accepted timestamp path or normalize Z compatibly, and require the full matrix to rerun green.

2. The new cancellation-successor validator fails open. Its context check is conditional on isinstance(payload.get("cancellation"), dict), so deleting cancellation from both the request and delivery is accepted. Require the context on both events and bind the request/root/lineage identities to the successor and to each other.

3. The request/delivery state machine is not exact. setdefault(kind, command) accepts duplicate requests or deliveries when the command id repeats, and either half can be absent. Require exactly one request followed by exactly one delivery, the same command/context, and nondecreasing request/delivery time. Include deletion, duplicate, mismatched-context, and delivery-before-request counterfactuals.

4. A delivery can name an unrecorded successor step. Changing the delivery sequence from 3 to 4 passes because the only check is step >= resume, although no step-4 cancellable command exists. Bind delivery to an actually recorded successor cancellable step (the step-3 activity in this evidence); test an unrecorded later step as well as reused steps.

Each counterfactual should pass the current validator and fail the corrected one. The Release-body edit remains unapplied and separate. Integration is still running, but this head is already non-qualifying because of the 3.10 failure.

[rmcdaniel]

General interim review (Security unavailable; operator instruction recorded at cloud#293 6617758869): exact head 52940fb7522b632e9d303755ae4e5621ec8e81a3: HOLD, narrow.

Reviewed by an independent reviewer, not the authoring agent.

Confirmed:

• Scope: tests, fixtures and CHANGELOG only; all 77 existing replay-regression blobs are identical to main; diff --check is clean.
• All four items of Security hold 6617712901 are closed, reproduced by mutation on both cancellation successors. Each mutation was accepted by bd41be60 and is refused here:
  • Python 3.10 trailing-Z times (the CI 3.10 job ran the 96 evidence tests green);
  • context missing from both events, the request only or the delivery only;
  • a duplicate or half-only request/delivery;
  • a delivery before its request, by time or by sequence;
  • a delivery to an unrecorded step (4 or 99).
• Registry gate: it selects exactly the 9 successors plus the SHA-pinned non-qualifying redrive-start-replayed-clock (70653171…, fails "history order"). A main-versus-head sweep of all 72 history fixtures changes only the 3 new successors.
• Non-leakage: the probe and positive control are sound. The cancellation clock is first observed at the delivery, and the validator forces reused times ≤ start ≤ every successor event. The CHANGELOG and proposed text make no behavioural overclaim.
• Release hashes reproduce: current 5d90b2f9…, proposed db167b0e2e36fa02…. The release is not edited.
• CI: 21/21 success, CLEAN. Locally, the 4 target files gave 565 passed.

Blocking, F1: the copied TimerScheduled skips run binding.

• Where: tests/test_replay_clock.py:588 (continue) runs before the workflow_run_id checks at :596/:611.
• Repro: on redrive-successor-reused-timer, setting the copied schedule's workflow_run_id to "bogus-run" is accepted.
• Why it's wrong: Core 28893f97 FailedRunRedrivePlan::successorEvents() (:437-455) copies the source scheduled_payload and changes only timer_id, and the accepted sdk-php#249 rule refuses any run other than the source.
• Required: workflow_run_id must be absent or equal to continued_from_run_id, with bogus-run and successor-run counterfactuals.

Also required, cheap and Core-aligned: refuse a copied schedule that carries timer_kind or cancellation_scope_id, because Core's isPlainTimer (:497) excludes those timers.

Non-blocking, tracked as a cross-SDK follow-up: validator strictness gaps shared with sdk-rust and sdk-php:

• parent_request_id skips the root check;
• only lineage[-1] is checked;
• a delivery to a step that already completed is accepted;
• command_type, call_kind, requested_at and schema are unchecked.

The release body stays unedited until source clearance and a separate exact-text go.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rmcdaniel]

General interim review (Security unavailable): exact head 262eb6ef1352d1ff4eb494bcd744ec4bb05ce517 is CLEARED for merge.

The delta from the reviewed 52940fb7 touches 2 test files only (tests/test_replay_clock.py, tests/test_redrive_successor_evidence.py). It closes F1:

• _copied_timer_schedule now refuses a copied TimerScheduled whose workflow_run_id is present and is not continued_from_run_id;
• it refuses any non-null timer_kind, cancellation_cleanup, cancellation_scope_id, parallel_group_id, condition_wait_id or signal_wait_id. That is exactly Core FailedRunRedrivePlan::isPlainTimer (28893f97 :497-511), checked against the source.

Counterfactuals on both timer successors: the bogus-run, successor-run and six plain-timer-key cases (16) were accepted on 52940fb and are refused here. The source-run and null-key controls are accepted. The 72-fixture verdict sweep still changes only the 3 new successors.

Hosted CI: 20/20 success, including Python 3.10, CLEAN.

The 2.13.1 Release body edit is a separate step: recompute the hashes, then a body-only PATCH and readback.

[rmcdaniel]

Merged and 2.13.1 Release corrected (General interim, Security unavailable).

Merge: 09bf1122ac10aa63066ba44f18d7fe4745b5b84d, parents main 7dbeca44 and the cleared 262eb6ef. The merge tree aac46f308e8d equals the head tree. #237 is closed.

2.13.1 Release body: a body-only PATCH appended the reviewed correction paragraph to the raw body, with one blank line and a final newline.

• Before: raw sha256 5d90b2f916630c6224ec319e5cf403e30a29b9177a4ef3da38795b9eafca17c0.
• After: raw sha256 db167b0e2e36fa0294aa6879e654dc0f442262bd68f0099c25c6327649ef678f, byte-equal to the uploaded body.
• Metadata is identical before and after (id 433065037, tag 2.13.1, target main, not a draft or prerelease, 0 assets, published_at 2026-11-03T18:22:41Z).
• releases/latest is still 2.13.1, and the tag ref is still object fba27c38.