Skip to content

Replay a condition wait whose orphaned duplicate's timer fires - #254

Merged
rmcdaniel merged 3 commits into
mainfrom
fix/orphaned-duplicate-condition-wait
Nov 4, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
fix/orphaned-duplicate-condition-wait

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes #253. This is the Python counterpart of sdk-php#283, using the rule General set for it.

Bug

Server 3.0.x can record a duplicate open_condition_wait while the first wait is still open (server#1015 / sdk-php#269 b). The first wait's timer stays live. Here is the sequence that breaks:

  1. a wrong signal satisfies and reopens the newer wait;
  2. later, the orphan's condition_timeout fires;
  3. Python replay binds the first wait_condition yield to the orphan and honours its timed_out resolution before considering the duplicate.

Workflow code takes the timeout branch while history continues with the reopened chain, so replay raises NonDeterministicReplayError. Every Python release tested has this: 2.12.4, 2.13.1, PyPI-installed 2.13.2, and main d5e31249. It is not a 2.13.2 regression.

Fix (workflow.py)

  • The rule: a wait counts as reopened when the next same-logical open (same key or fingerprint) was recorded before the earlier wait's terminal event. That is the state when the next wait opened. In that case the earlier wait's later timed_out doesn't end the logical wait, and replay continues to the next open.
  • The grace vector stays separate: a wait that timed out before the next same-key open is still finished, and the next open is its own call.
  • Bookkeeping: wait_opened_event_indexes records each open's history index, and _opened_before_terminal compares it with condition_wait_terminal_indexes.

Evidence

  • TestOrphanedDuplicateWait, sample-app's shape ported to Python: w3 and w4 duplicate opens of 300 s, a wrong signal at +120 s, w4 Satisfied, and a w5 reopen.
    • After the orphan fires, replay reopens with 0, for both reopen shapes (300 as older SDKs write it, 180 as 2.13.2 does). The first open's deadline has passed, and replay doesn't throw.
    • Before the orphan fires, replay reopens with 180.
    • When w5's own timer then fires, the workflow takes its timeout branch (expire-proposal).
  • Fail-before on main's source: the 2 orphan-fired tests fail, and the controls pass.
  • Grace controls from Pin the condition-wait reopen rule after a timeout and in selections #250 (late, 31 and 271) still pass.
  • Corpus fixture condition-wait-orphaned-duplicate-timer-fired expects open_condition_wait with timeout 0. It passes here and fails on main's source.
  • The neighbouring suites pass: durable selection, update/signal/condition replay, condition replay drift, and the wait-condition tests (191 tests).

Interaction with #251

#251 (now at a66bdc9a) applies the reopen bound only when an open is linked as a reopen, which it decides by resolution != "timed_out". With this fix, an orphan whose later timeout is superseded is also a linked reopen, so the bound should apply there too. Whichever PR merges second will route the bound through the same "linked" decision. I'll do that refresh with a test.

Local runs at 6577584 (lab2, Python 3.13, this branch's source)

  • ruff is clean, and mypy src/durable_workflow reports no issues.
  • Full unit suite (excluding tests/integration): 4918 passed, 4 skipped.
  • validate-regression-corpus.py --base-ref origin/main: status: pass.

Not merged or released.

🤖 Generated with Claude Code

Server 3.0.x can record a second open of a condition wait while the
first is still open (server#1015). When the first open's timer fired
later, replay ended the logical wait with that timeout while history
continued with the reopened wait, raising NonDeterministicReplayError.
A wait now counts as reopened when the next same-logical open precedes
the earlier wait's terminal event; a wait that timed out before the next
open stays finished (the grace vector). sample-app's sdk-php#283 shape
is a corpus fixture.

Fixes #253.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel rmcdaniel added the agent:sdk-python Work by the sdk-python repo agent label Nov 4, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

General interim review (Security unavailable): content CLEAR at b712d01dd8b1fadc3275d775bc3aa134848d92ab (independent reviewer). Merging is gated on the refresh below.

  • Verified on main: the 3 orphan-fired tests and the condition-wait-orphaned-duplicate-timer-fired fixture fail there and pass here, and the before-fire controls pass on both. Full suite 4922 passed. CI 21/21.
  • About 30 probes, all correct:
    • Grace stays separate: a plain timeout, a grace wait after orphan-chain expiry, and TimedOut recorded before the next open with TimerFired after it.
    • A chain of 5 opens, and multiple orphans.
    • Keyless waits with a fingerprint.
    • The legacy sequence-only TimerFired, and ConditionWaitTimedOut as the terminal event.
    • PHP-server and native histories behave the same (the rule doesn't use occurrence ids). Grouped and selection waits are never linked.
    • Live: the timeout is never negative. When the server opens a 0 s wait and it fires, the expiry activity follows, then completion. Nothing loops.
    • The orphan-fires-before-any-wake shape raised NonDeterministicReplayError on main, so no recorded Python history changes outcome.

Required before merge (after #251 merges first): refresh onto main and keep both sides of the textual conflicts: the adjacent helpers in workflow.py, CHANGELOG, and the end of test_wait_condition.py. Then call _assert_reopen_within_first_timeout(first_open_index, next_wait_index) before consumed_reopen_for_current_wait = True in the new orphan or timed-out branch (workflow.py:7713-7720).

Without it, this tampered history replays to Wait(0) but is refused before w3 fires: w3 300 s, duplicate w4 600 s or no timeout, wrong signal on w4, w5 180 s, w3 TimerFired. Add:

@pytest.mark.parametrize("duplicate_timeout", [600, None], ids=["longer", "dropped"])
@pytest.mark.parametrize("orphan_fired", [False, True], ids=["before-orphan-fires", "after-orphan-fires"])
def test_an_orphans_duplicate_open_is_bounded_like_any_reopen(duplicate_timeout, orphan_fired):
    history = _orphaned_duplicate_history(180, orphan_fired)
    duplicate = history[3]["payload"]
    assert duplicate["condition_wait_id"] == "w4"
    if duplicate_timeout is None:
        del duplicate["timeout_seconds"]
    else:
        duplicate["timeout_seconds"] = duplicate_timeout
    with pytest.raises(NonDeterministicReplayError, match="condition_wait_reopen_timeout_exceeded"):
        replay(OrphanedDuplicateDecision, history, [])

Verified: 2/4 fail without the line and 4/4 pass with it. The full combined suite passes (4942).

Follow-up issue, not this PR: an orphan ended by ConditionWaitSatisfied recorded after the duplicate opened, with the predicate true, still raises NonDeterministicReplayError (workflow.py:7732-7744). This is also the case on main. The 3.0.5 server wakes only the newest wait, so this is hypothetical today.

Keep both sides of the additive conflicts (the workflow.py helpers, the
Unreleased Fixed entries and the test file tail). The orphan branch now
calls _assert_reopen_within_first_timeout before consuming the
duplicate as a reopen; test_an_orphans_duplicate_open_is_bounded_like_any_reopen
covers longer and dropped duplicates before and after the orphan fires
(the two after-fire cases are accepted without the bound).

Refs #253.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

General interim review (Security unavailable): CLEAR at 669a4cb.

Merge at this exact SHA and post a readback (merge tree = head tree). Then open the 2.14.0 release prep with #252, #251 and #254.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 4, 2026 04:58
@rmcdaniel
rmcdaniel merged commit e319c9b into main Nov 4, 2026
21 checks passed
@rmcdaniel
rmcdaniel deleted the fix/orphaned-duplicate-condition-wait branch November 4, 2026 04:58
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged at General's clearance of exact head 669a4cbe1795385ac7332a38b98844bd8edda4bc (21/21 green).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:sdk-python Work by the sdk-python repo agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replay a condition wait whose orphaned duplicate's timer fires

1 participant