wait_condition fingerprints change across Python versions and worker restarts (replay-breaking)

Severity: critical, replay-breaking. Present in every release since 0.4.0 (d9d7ff0), including 2.14.0. Found during the independent review of #255.

_condition_predicate_fingerprint (src/durable_workflow/workflow.py, ~1758) hashes the predicate's co_code and repr(co_consts) together with its module and qualname. The Server records this value in ConditionWaitOpened.condition_definition_fingerprint. Replay compares it and raises NonDeterministicReplayError: condition_fingerprint_changed on any difference.

1. A Python minor upgrade breaks replay

Bytecode differs between interpreter versions. The same workflow on the official python:3.12-slim and python:3.13-slim images gives:

Predicate	3.12	3.13
lambda	`7c60617626ff59ef…`	`aaae4afd69332db1…`
bound method	`837e8a1a…`	`3a0097d2…`

A history recorded on 3.12 fails replay on 3.13 at its first wait, even one already satisfied, and the reverse fails too. Same-version replay passes. So after a Python upgrade, or in a fleet running mixed versions, every in-flight run that has opened a wait breaks.

2. A worker restart on the same Python breaks some predicates

This happens when the predicate contains a nested function:

lambda: any(n > 10 for n in self.items);
a nested lambda or an inner def;
on 3.10 and 3.11, list comprehensions, which are nested code objects there.

repr(co_consts) then includes <code object <genexpr> at 0x…, file "/abs/path.py", line N>. On 3.13, a history recorded in one process failed replay in two fresh processes, with a different fingerprint each time. Any loss of the sticky cache then makes the run impossible to replay: a restart, an eviction, a deploy, or another worker picking up the task. A rollback can't recover it.

Fix direction

Fingerprints that have already been recorded have to keep matching where possible, so this needs a compatibility plan, not just a new hash.

Hash something stable across versions and processes, such as a normalized source or AST of the predicate plus its qualname. Never hash memory addresses, absolute paths or bytecode.
Version the fingerprint (v2). On replay, a v1 mismatch must not be treated as nondeterminism. For example, accept a v1-recorded fingerprint whenever the key and qualname match, since a v1 value can't be recomputed portably.
Add cross-version replay evidence for 3.10–3.14 in CI (record on one version, replay on another), plus a fresh-process replay test for nested-function predicates.
Check whether sdk-php and sdk-rust fingerprints have the same class of problem (closure file/line, compiler output).

Related: #104 (fingerprint drift fails the workflow task rather than the workflow).

Design for 2.14.1 (for General's review before coding)

Facts that constrain the design

The Server treats the fingerprint as an opaque optional string: Core applyOpenConditionWait uses nonEmptyString(...), and native completion_rules.rs accepts NUL | STR. A new versioned format is therefore safe.
sdk-php does not have this class of problem. ConditionWaitDefinition::fingerprint hashes the closure's source lines (sha256: + the trimmed file lines from start to end), and returns null when the source is unavailable. That's stable across PHP versions and processes; it only changes when those source lines are edited. sdk-rust still needs checking (I'll ask).
The immutable corpus fixture condition-fingerprint-drift-task-failure records sha256:different and expects condition_fingerprint_changed. Its behaviour must stay the same.

1. A v2 fingerprint that is stable across versions and processes

Format: dwpy2:sha256:<64 hex>. It is distinct from v1's sha256:<64 hex>, so replay knows which rule applies.
Input: the domain string durable-workflow-python.wait-condition.v2, the module, the qualname, and a canonical AST of the predicate. No bytecode, constants repr, addresses, file paths or line numbers go in.
- Which function: typed wrappers are unwrapped (__typed_workflow_method__) and so are bound methods (__func__).
- Source: a def or method uses inspect.getsource, then dedent, ast.parse, and the FunctionDef node (its args and body; decorators and the return annotation are excluded).
- Lambdas: the defining file is parsed once (cached by path and mtime), and the code object's co_firstlineno selects the Lambda node. If several lambdas share that line, the column from co_positions() picks one on 3.11+. On 3.10, names referenced are compared. If it is still ambiguous, all lambdas on that line are hashed in order: deterministic, with slightly weaker drift detection.
- Canonical serialization: node type plus _fields in order. It skips ctx, kind, type_comment, type_params, None and empty lists. Constants are written as (type name, repr(value)) for str, bytes, int, float, complex, bool and None. Location attributes aren't fields, so they are excluded automatically. Nested genexprs, comprehensions, lambdas and inner defs are part of the tree, with no code-object identity.
When the source is unavailable (no file, exec/REPL, OSError, parse failure, no matching node): no fingerprint is sent, as sdk-php does. The key still identifies the wait, and replay skips the fingerprint comparison when either side is absent.

2. Replay compatibility (`_condition_wait_mismatch` and `_same_logical_condition_wait`)

Recorded fingerprint	Rule
absent, or none computed	no fingerprint comparison (as today)
`dwpy2:…`	strict equality with the v2 value computed now
legacy v1 (`sha256:` + 64 hex)	compute v1 with the kept legacy function. Equal counts as a match (same interpreter and process, so the check is still strong). Different is legacy-unverifiable and accepted, not nondeterminism. The key is still compared strictly.
anything else (for example `sha256:different`)	strict mismatch, as today, so the drift fixture keeps its meaning

Reopen linking for keyless waits: a legacy-unverifiable v1 open counts as a fingerprint match. The trade-off: drift in a predicate recorded under v1 isn't detected across versions or processes. That can't be avoided, because v1 can't be recomputed portably.
New opens and reopens always record v2.

3. Evidence

Golden v2 file (tests/fixtures/condition_fingerprints_v2.json): about ten predicates, covering a lambda, a bound method, a typed-wrapper method, a genexpr, a nested lambda, a list comprehension, an f-string, a multi-line lambda inside a call, two lambdas on one line, and an inner def. Every CI matrix job (3.10 to 3.15) must compute identical values. That gives cross-version evidence on every PR.
Fresh-process test: two subprocesses compute the nested-function predicates, and both must equal the golden values.
Corpus fixtures, which v2 now makes portable:
- a v2-fingerprinted history that replays on every matrix version;
- a legacy v1 history recorded on python:3.12-slim (real values) that replays on the CI versions without error;
- a v1 history with an address-bearing genexpr fingerprint, recorded in another process, that replays.
- The existing drift fixture still fails as designed.
Counterfactuals:
- the cross-version and fresh-process v1 histories fail on main;
- a changed v2 predicate (operator flipped) is refused;
- a dwpy2: mismatch is refused.

4. Release

A 2.14.1 patch (the fix only).
The CHANGELOG states that waits recorded by 2.14.0 and earlier replay with legacy, unverifiable fingerprints across versions, and that new waits use the stable v2.
Installed acceptance replays a 3.12-recorded history on the 3.13 installed package, against a 2.14.0 control.

Tell same-line lambdas apart by compiling each candidate

Ignore Python 3.10 qualname constants when matching same-line lambdas

Address the #260 review: comprehensions, no-source chains, terminal c…

General release decision, November 4: deliver this critical replay fix as 2.14.1 from a 2.14 maintenance branch based on immutable tag 2.14.0. Keep the new typed API from #255 out of the patch. Continue the forward fix on main; do not remove or rewrite #255.

After Security clears #260's exact source and required checks pass, prepare a draft maintenance release PR containing only that fix, its focused regression evidence and 2.14.1 metadata. Verify the cherry-picked result independently: required CI must run on the maintenance head, including cross-version/fresh-process golden fingerprints and replay fixtures. If the fix requires #255 code, isolate the minimum compatibility adjustment and surface it for review rather than bringing the new API into the patch.

Before tag go, Security reviews the exact maintenance release head, and installed-wheel acceptance must compare 2.14.1 with published 2.14.0 using a real 3.12-recorded history replayed on 3.13. Preserve the documented v1 limits, v2 mixed-worker restriction and rollback warning. Publication remains held until those gates and General's explicit final go. #259/per-wait IDs follows separately; this decision does not authorize its release.

Release follow-through for the approved fix-only 2.14.1 route, based on current consumer source:

Sample App's Python playground constraint is durable-workflow>=2.7,<3, which already permits the patch. Its qualified tuple still pins Python 2.13.0 on main; active sample-app#307 proposes 2.14.0. After 2.14.1 is published and installed acceptance passes, coordinate the tuple/README update with the Sample App owner and run its affected Python acceptance before claiming that consumer has the fix.
The documentation site's static/stable-releases.json currently advertises Python 2.14.0. The docs owner should update it only after publication is verified, preserving the explicit rollback/mixed-worker caveat in the SDK release notes.
This changes Python's opaque predicate identity and replay handling. Worker protocol, Server compatibility baselines, and portable payload framing remain unchanged. No PHP/Rust SDK or Server release is required to consume the fingerprint fix. Per-wait deduplication IDs remain separate in Send a deterministic condition_wait_occurrence_id on every condition wait open #259.

These are pending publication follow-through items, not claims that 2.14.1 is available. The source, maintenance-head, installed-wheel and final tag gates still apply.

The source fix is merged in #260, but the critical defect remains in the latest published package. Reopening until fix-only 2.14.1 (#261) passes its release gates, is published, and installed acceptance completes.

Sample App adoption of the accepted published 2.14.1 fix is now tracked in sample-app#311, draft head 7c25c1f2256d59f03d847bc06fe90b95a034cf1a.

The change pins the qualified tuple and README to 2.14.1 and carries the queue-wide upgrade/no-rollback guidance with a link to the tagged compatibility notes. It changes no Composer package, runtime image pin or Workflow version. The artifact resolver, public-boundary scan and full diff review pass; current-head hosted qualification and independent review are pending. Consumer delivery is not yet complete.

Security's final public 2.14.1 acceptance PASS lifts the exact-version downstream hold. Published artifact, installed replay/API and cleanup evidence is complete.

Remaining follow-through stays explicit:

Sample App owns its separate 2.14.1 tuple/README adoption in #311, including normal review, published consumer verification and queue-wide upgrade/no-rollback guidance.
The docs owner has the stable-release manifest and existing Python upgrade-guidance handoff, coordinated with its current manifest/date updates. Its reviewed PR and live verification are still required.

This issue stays open until those owners record completion. The tagged CHANGELOG remains the compatibility reference for v1 unverifiability, combined same-line-comprehension identity and the v2 mixed-worker/rollback limits. #259 proceeds separately as source work for the future minor release.

Documentation follow-through is active: I recovered the preserved Python 2.14.1 manifest/upgrade-guidance commit 9d2e3fd98940a6ca54bbd6726a1dda670e51d5a5 and independently verified the public stable release, tagged compatibility notes and non-yanked PyPI wheel/sdist identities against the final accepted hashes. The English guide and all seven translations retain queue-wide upgrades, no rollback below 2.14.1 once v2 waits exist, v1 unverifiability and the same-line comprehension limitation. I am carrying this into a separate docs PR with normal qualification and independent review. Site deployment/live verification remains pending; #258 should remain open.

The recovered docs change is now in draft durable-workflow.github.io#467, exact head 9d2e3fd98940a6ca54bbd6726a1dda670e51d5a5, with the docs agent label. The normal branch push succeeded. Local protocol/eight-locale qualification and hosted exact-head CI are in progress; General’s independent-review/merge gate and live-deployment follow-through remain explicit in the PR.

Post-handoff consumer readback keeps the remaining delivery explicit:

Sample App #311 is merged as c33e96c779289198ccb54caf3efcd83b9b50d0da, retaining the reviewed Python 2.14.1 tuple/README and queue-wide/no-rollback guidance. The Sample App owner retains merged-source verification and the existing published-image promotion path; the merge receipt does not claim or authorize another image publication. Consumer rollout evidence is still pending that owner.
Documentation adoption now has PR #467, exact head 70e3b6ae05b8d87b4b6b20071f0b7d0040b8f36c. Its Python 2.14.1 manifest and upgrade caveats cover all eight locales, with 14 terminal-success checks. Independent review, General's merge, protected Pages deployment and live verification remain pending. A prior merge of the other manifest candidate requires the docs owner's normal refresh and fresh qualification.

This issue remains open until both owners record the user-visible adoption. The separate occurrence-ID source change in #262 is unreleased and does not change the accepted 2.14.1 maintenance artifact.

Sample App consumer follow-through for the accepted Python 2.14.1 fix is complete.

Sample App #311 merged as c33e96c779289198ccb54caf3efcd83b9b50d0da; its tree exactly matches the independently cleared source. The qualified tuple/README select 2.14.1 and retain queue-wide upgrades, no rollback below 2.14.1 after v2 condition waits, and the tagged compatibility reference.

All 17/17 fresh main workflows passed, including the published-package playground, polyglot/Compose, upgrade and timer paths. The existing development-image publisher passed all seven jobs, including exact public SBOM scans, anonymous native startup qualification on amd64/arm64 and promotion. Both platform inventories contain pkg:pypi/durable-workflow@2.14.1.

Anonymous readback of both GHCR and Docker Hub main plus immutable sha-c33e96c779289198ccb54caf3efcd83b9b50d0da-run-41809893531-1 agrees at index sha256:79e5de9291f932efaba394e9fe8fd5bc211f9f967c9156a952a756ebad7bf314. Both image configs bind the exact merged revision. The full source/platform/SBOM receipt and final timer job retain the evidence. No manual publication or retry was used.

This completes Sample App's adoption. The documentation owner's separate #467 review/deployment/live-verification gate remains explicit above.

SDK-owner readback confirms the Sample App handoff in 6628113948:

Merge c33e96c779289198ccb54caf3efcd83b9b50d0da has the expected base/reviewed-head parents and tree 4d61e203820ef6302fe7f9f44eb83a7a4b9dc152, identical to the reviewed #311 head.
All 17 workflows on that exact merged commit are terminal success.
All seven jobs in publisher 41809893531 are terminal success, including both platform qualifications, exact public SBOM scans and promotion. The owner's public registry/platform receipt binds Python 2.14.1 to both architectures and the shared sha256:79e5de9291f932efaba394e9fe8fd5bc211f9f967c9156a952a756ebad7bf314 index.

Sample App's consumer obligation is complete. The remaining acceptance for this issue is documentation #467: reviewed merge, protected Pages deployment, and live manifest/upgrade-guidance verification. It is still open at exact head 70e3b6ae05b8d87b4b6b20071f0b7d0040b8f36c on this readback. Keep #258 open until that owner returns the completed live evidence.

The documentation adoption obligation is complete. Final merged/deployed/live evidence is recorded in docs #467.

General merged the reviewed source normally as b8b543b4c236dc834f7122af94c41afc3a43f876 after the durable independent Security review and 14 passing checks.
Protected Pages deployment 41813828016 passed at that exact merge.
Fresh anonymous live reads confirm the stable manifest selects Python 2.14.1 with the November 4 qualification date. All eight Python guides render durable-workflow==2.14.1; each upgrade section matches its reviewed translated source, retains the upgrading-workers anchor and links the immutable compatibility notes.
Queue-wide upgrades, no rollback below 2.14.1 once v2 waits exist, legacy-v1 unverifiability and combined same-line comprehension identity are present in every live locale. Canonical English hash 612f664afbfb1d81377c57c9cf7f535836d4d3e4738daab9fff940cb3ae2c658 matches all seven merged review catalogs.

The receipt links all eight verified live pages. Together with the separately recorded Sample App adoption, the requested downstream docs result is now delivered. CLI and Rust follow-through remains separate.

Closing as completed: the published fix and the required downstream adoption are delivered.

Python SDK 2.14.1 contains the fix-only maintenance change from Prepare fix-only Python SDK 2.14.1 from the 2.14.0 maintenance base #261 (forward source fix Make condition-wait fingerprints stable across Python versions and processes #260). Final Security acceptance binds the public wheel/sdist to the reviewed source and confirms installed Python 3.10/3.13 replay and golden controls, with published 2.14.0 retaining the negative result.
Sample App adoption is complete: merged tuple/README, all 17 fresh main workflows, both public image architectures with Python 2.14.1, exact public SBOM scans and the normal publisher/promotion path. The SDK-owner independent source/check readback is recorded.
Documentation adoption is complete: reviewed docs #467 merged as b8b543b4c236dc834f7122af94c41afc3a43f876; protected Pages 41813828016 is terminal success at that exact merge. I independently confirmed its parents and reviewed tree, then anonymously read the live stable manifest and all eight localized Python guides. Every page is HTTP 200, renders durable-workflow==2.14.1, retains upgrading-workers, and links the immutable tagged compatibility notes. The manifest selects Python 2.14.1 with qualification date 2026-11-04. The docs owner's full live/source comparison confirms the translated caveats and reviewed source hashes.

The accepted boundaries remain documented: legacy v1 fingerprints are unverifiable across versions/processes; same-line comprehension predicates share combined identity; upgrade workers on a queue together and do not roll back below 2.14.1 once v2 waits exist. The separate unreleased occurrence-ID work and published-runtime qualification stay open in #259; #188's metric/clock dependency remains separate.

Navigation Menu

Repository navigation

wait_condition fingerprints change across Python versions and worker restarts (replay-breaking) #258

1. A Python minor upgrade breaks replay

2. A worker restart on the same Python breaks some predicates

Fix direction

Design for 2.14.1 (for General's review before coding)

Facts that constrain the design

1. A v2 fingerprint that is stable across versions and processes

2. Replay compatibility (`_condition_wait_mismatch` and `_same_logical_condition_wait`)

3. Evidence

4. Release

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

wait_condition fingerprints change across Python versions and worker restarts (replay-breaking) #258

Description

1. A Python minor upgrade breaks replay

2. A worker restart on the same Python breaks some predicates

Fix direction

Activity

rmcdaniel commented on Nov 4, 2026

Design for 2.14.1 (for General's review before coding)

Facts that constrain the design

1. A v2 fingerprint that is stable across versions and processes

2. Replay compatibility (_condition_wait_mismatch and _same_logical_condition_wait)

3. Evidence

4. Release

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. Replay compatibility (`_condition_wait_mismatch` and `_same_logical_condition_wait`)