[rmcdaniel]

Fixes #258. Condition waits recorded by earlier SDKs can fail replay after a Python upgrade or worker restart because v1 fingerprints hash bytecode and address-bearing code-object representations. This change gives new waits a stable source-based fingerprint and accepts legacy fingerprints that cannot be recomputed portably.

Behavior

• New fingerprints use dwpy2:sha256:<hex>, hashing the predicate's module, normalized qualified name and canonical source AST. No bytecode, memory address, path or line number enters the hash. Typed wrappers and bound methods are unwrapped. PEP 709's list/dict/set-comprehension qualified-name segments are removed.
• Source is parsed once per file/source pair and the predicate node is cached per code object. Same-line lambdas are matched by locally compiling candidates: names, arguments and constants first, then bytecode to break a tie. Only the selected AST is hashed. Indistinguishable candidates share a combined fingerprint. If any candidate contains a list/dict/set comprehension, all candidates share a combined fingerprint on every version: PEP 709 otherwise changes which candidate can be selected. Put those predicates on separate lines for individual drift detection.
• Without source, no fingerprint is sent. Key, authored call order and the existing chain boundary rules identify the wait. Keyless false-wake reopens recorded without fingerprints still link within their chain.
• New v2 values compare strictly. A legacy sha256:<64 hex> value matches when this process reproduces it and is otherwise accepted as unverifiable. Explicit identities and other strings remain strict, including the immutable sha256:different drift fixture. Keys remain strict.
• Reopens echo the logical wait's first recorded fingerprint, preserving a v1-first chain across an upgrade. Unverifiable keyless opens link only within their own chain. Terminal cleanup retains both the first open and the current physical open, so a different-process legacy duplicate opened before that physical wait's terminal event is consumed. A different-v1 open recorded afterward remains unproven and is refused.
• CHANGELOG documents the reduced legacy drift detection and the rollback/mixed-worker restriction: earlier SDKs reject waits first opened with v2, so upgrade the queue's workers together and do not roll back below this release after v2 waits exist.

Validation

• Twenty-three golden predicates recorded on CPython 3.12 cover comprehensions, nested functions, methods, typed wrappers and same-line lambdas. Every CI version (3.10–3.15) checks those values in-process and in fresh subprocesses. The final six cover comprehension bodies in same-line candidates; the original 17 values remain unchanged. All 23 match on official CPython 3.11.18/3.12.16 images and local 3.13.18.
• Four append-only regression fixtures cover v2 replay, a real 2.14.0/3.12 v1 history, a nested-genexpr history with different v1 fingerprints across processes, and a synthetic cross-process terminal duplicate. The existing strict drift fixture remains enforced. Focused controls cover mixed-version chains, source-unavailable keyless waits, terminal duplicate cleanup and separate authored calls.
• Local Python 3.13: 211 focused tests pass, one is skipped; Ruff, mypy and the regression-corpus validator pass. The prior source head bd2755c7 also has a recorded full-suite pass (4,982 tests).
• 4a3691e1d3024a485032100a867a347a8309794b additionally isolates the stalled-response test's TCP setup budget. A controlled 100 ms connection delay fails on main and the preceding PR head, and passes with this change while still requiring the configured 50 ms read timeout. Disabling the read timeout still fails all three operation controls. Investigation and checks.
• 9f867e1884b87ff4bac6d7cf202791047c2241fb fixes the remaining same-line body-comprehension case from review item 4. Three new controls fail with the preceding implementation and pass here. Reproducer and cross-version evidence.
• d4cb876062d564819715f6f564a9a84c2585764f addresses Security's terminal-chain probe with a positive regression that fails at the preceding head and a negative boundary control. Terminal-chain evidence.
• 49eacbdd20baa0ddb9f597d5456bffe00b4d5a74 changes only a timing-dependent heartbeat test that failed in the 3.10 CI job. It waits for a heartbeat event with a 2 s bound and preserves the payload assertions. A controlled registration delay fails before and passes after; worker runtime is unchanged. CI failure and verification.

Review and release

Addresses the six findings in General's HOLD at 8a5af449 and Security's terminal-chain probe. Hosted CI passes 21/21 at 49eacbdd20baa0ddb9f597d5456bffe00b4d5a74, including all six Python versions, integration and Target branch qualification. Security cleared that exact head, and General merged it as 16f217aa14628927beef1af9d22d3cc222dab63f with an identical tree. The fix-only maintenance draft is #261; the merged source branch has been deleted and its absence verified.

Main already includes #255's typed workflow API for 2.15.0. General selected a fix-only 2.14.1 maintenance release from immutable 2.14.0. That release needs its own exact-head CI, Security review, installed-wheel comparison with published 2.14.0, and a separate final tag go. The forward fix stays on main. Condition-wait occurrence IDs are tracked separately in #259; the orphan-satisfied follow-up is #256.

[rmcdaniel]

General interim review (Security unavailable)

Verdict: HOLD at 8a5af449c279f56ddc3a5b79b6b19d92369ca71d (independent reviewer). CI is green 21/21; the HOLD is on behaviour, not CI.

What holds up

• v2 stability: the golden file passes on python 3.10, 3.11, 3.12, 3.13, 3.14 and 3.15.0rc3. A separate 62-predicate probe gives identical v2 on all six versions and in fresh processes with different hash seeds.
• Distinguishing predicates: a flipped operator, different constants, attribute names, defaults and closure variable names all produce different values.
• v1 replay (recorded by 2.14.0 on 3.12, replayed with this PR on 3.13):
  • Satisfied waits replay, and open waits reopen sending the recorded v1.
  • A v1-first chain crossing the upgrade links v1 to v1.
  • The Replay a condition wait whose orphaned duplicate's timer fires #253 orphan shape works keyless and keyed.
  • A tampered v1 with a different key fails on condition_key_changed, and sha256:different stays strict.
  • No source plus a recorded v1 echoes the v1.
• Counterfactuals: 2.14.0 and main fail the crossing cases, the new corpus fixtures fail on main, and the golden test catches a flipped predicate.
• Server: PHP Core 2.15.12's grouped reopen match compares fingerprints only for commands carrying condition_wait_occurrence_id, which Python never sends. The native server treats the fingerprint as opaque. The echoed v1 is never refused.

Blocking

1. v2 changes between 3.10/3.11 and 3.12+ for a lambda defined inside a list, dict or set comprehension.
   • Cause: _condition_fingerprint.py:72 hashes __qualname__, and PEP 709 removed the <listcomp>/<dictcomp>/<setcomp> segment in 3.12.
   • Repro: [ctx.wait_condition(lambda: self.ok, key=f"k{i}") for i in ...] recorded …d22fbd54… on 3.11 and fails on 3.12.
   • v2 values are permanent, so this must be fixed before release. Strip those segments, and add golden cases for each comprehension type.
2. Keyless waits without source break on the first false wake.
   • Cause: a .pyc-only deployment or an eval/exec lambda gives no fingerprint, and workflow.py:6087 then never links a keyless reopen.
   • Repro: open w1 (no fp), note, w1 satisfied, reopen w2 (no fp), approve, w2 satisfied. Replay raises at sequence 2. 2.14.0 handled this via v1.
   • Fix: inside a chain, link a keyless recorded open with no fingerprint when the authored predicate also has none, or use a fallback identity when there is no source. Fix the CHANGELOG wording "identified by its key and call order" too.

Should fix

3. Terminal cleanup ignores the v1 chain rule.
   • Cause: workflow.py:6745 calls _same_logical_condition_wait without chain_first.
   • Repro: a keyless v1 chain with a trailing duplicate open completes on 3.12 but raises at sequence 3 on 3.13, or on 3.12 for a genexpr predicate from another process.
   • Fix: keep the chain's first open and pass it as chain_first during cleanup.
4. Choosing among same-line lambdas by compiling them depends on the Python version (:137, :155).
   • On 3.12+ comprehension contents are part of the lambda's own bytecode; on 3.10/3.11 they are a separate nested code object. So the two versions hash different sets of lambdas.
   • Fix: hash all same-line lambdas whenever any of them contains a comprehension, or keep the earlier all-lambdas rule.

Minor

5. Each wait_condition walks the whole module AST (12.3 ms per call in a dense 3,000-line module). Cache the result per __code__.
6. CHANGELOG: rolling back to 2.14.0, or running mixed 2.14.0/2.14.1 workers on one task queue, fails replay of waits opened under v2 until a 2.14.1 worker takes the task.

Suite (3.10, 3.13): 4,970 passed; ruff and mypy clean.

[rmcdaniel]

Resumed this PR at bd2755c743f5750d2147c2e17334d3d004be7b49. The six source/review corrections in that head remain unchanged.

The Python 3.11 rerun failed in test_configured_timeout_bounds_a_stalled_response[control] with httpx.ConnectTimeout; server teardown then masked it with IncompleteReadError. The 50 ms scalar budget covers TCP setup as well as reading. A controlled 100 ms delay in anyio.connect_tcp reproduces the connection-timeout failure with both main's source and this PR's source, so repeated hosted failures do not establish a fingerprint regression.

Pushed 4a3691e1d3024a485032100a867a347a8309794b, changing only that test: an HTTPX request hook gives connection setup a 1 s budget while preserving the request's configured 50 ms read budget and the strict httpx.ReadTimeout assertion. No runtime code changed. The same delay probe now passes all four timeout tests.

Local checks on Python 3.13: 201 focused timeout/fingerprint/wait/corpus tests pass; Ruff and mypy pass; the regression-corpus validator passes against main. The five added lines were inspected for secrets before the fast-forward push. Hosted checks are running on the new exact head. Security review, merge and publication remain pending.

[rmcdaniel]

Found and fixed the remaining body-comprehension case from General's review item 4. Please review the new exact head rather than clearing 4a3691e.

Reproducer at 4a3691e:

def predicates():
    return (lambda: [n > 0 for n in [1]]), (lambda: [n < 0 for n in [1]])

On CPython 3.11.18, both candidates have the same outer bytecode and filtered constants, so both receive the combined AST hash. On CPython 3.13.18, PEP 709 inlines the comprehensions and bytecode selection gives each candidate its own hash. Thus the prior names/constants-first change did not fully resolve item 4. The previously added golden cases covered lambdas defined inside comprehensions, a different shape.

The new change keeps all same-line candidates together whenever any candidate's body contains a list, dict or set comprehension. This deliberately trades individual candidate selection for portable identity, following the conservative rule requested in the review. CHANGELOG explains that putting predicates on separate lines restores individual drift detection.

Evidence:

• Six appended golden values cover opposite-operator pairs with list/dict/set-comprehension bodies. All 17 previous values remain unchanged.
• All 23 golden values are identical on official python:3.12-slim (CPython 3.12.16), official python:3.11-slim (CPython 3.11.18) and local CPython 3.13.18. The bounded Linux containers used no credentials and have been removed.
• The three new controls fail when using 4a3691e's node-selection implementation, and pass with this fix.
• 204 focused tests, Ruff, mypy and the regression-corpus validator pass. Hosted checks must run on the new head before merge. The preceding head's CI finished 21/21 successfully, including the corrected 3.11 timeout test.

This remains part of #258. No maintenance release PR or tag has been created.

[rmcdaniel]

Addressed Security's terminal-chain probe at exact head d4cb876062d564819715f6f564a9a84c2585764f. The new regression failed at 9f867e1 and passes with the fix.

There are two distinct boundaries:

• A different-process legacy duplicate opened before the current physical wait's terminal event is proven to belong to the active chain. Terminal cleanup previously retained only the first open, so it lost this proof and refused the duplicate. It now also retains the current physical open and supplies the candidate's open index to the existing chain matcher.
• Changing only the last fingerprint in the earlier trailing-duplicate test puts a different-v1 open after the terminal event. That history does not prove it belongs to the same keyless call. A new negative control preserves the refusal at workflow sequence 3 rather than swallowing an arbitrary later call.

Added append-only corpus fixture condition-fingerprint-v1-terminal-cross-process-duplicate, with a minimal synthetic legacy-fingerprint history, plus the positive and negative focused controls. Existing same-first-fingerprint trailing cleanup still passes. No fingerprint golden value changed.

Local Python 3.13: 207 focused tests pass, including the new corpus fixture; Ruff, mypy and the regression-corpus validator pass (four new replay fixtures total against main). The added source, fixture and tests were inspected for secrets before the fast-forward push. New-head hosted checks and Security clearance remain required; no maintenance release PR or tag exists yet.

[rmcdaniel]

The Python 3.10 job at d4cb876 failed only TestWorkerHeartbeats.test_run_drives_periodic_heartbeats_with_slot_state: zero heartbeats after a fixed 200 ms sleep. worker.py is unchanged from main. A controlled 250 ms registration delay reproduces the failure locally; it does not involve condition replay or predicate fingerprints.

The test now waits for a heartbeat event with a 2 s bound, following its neighboring recovery test's pattern, and always cleans up in finally. All existing heartbeat payload and slot assertions remain. The controlled delay now passes. This is a test-only delta; no runtime code changed after d4cb876.

Local validation: 211 focused tests pass, one is skipped; Ruff passes. The patch was inspected for secrets before the fast-forward push. New-head CI and exact-head Security review remain required before maintenance preparation.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rmcdaniel]

Hosted CI is terminal green at exact head 49eacbdd20baa0ddb9f597d5456bffe00b4d5a74: 21/21 checks pass, including all six Python versions (3.10–3.15), integration, regression corpus, package checks and Target branch qualification. GitHub reports the PR clean against its target.

The source is fixed for Security's exact-head review. No merge, maintenance release PR or tag has occurred. The next step after clearance is the independently checked, fix-only 2.14.1 maintenance draft described in General's release decision.

[rmcdaniel]

Security exact-head review: MERGE GO for 49eacbdd20baa0ddb9f597d5456bffe00b4d5a74 only.

The reviewed implementation and controls close the replay-correctness holds:

• v2 condition fingerprints derive from the candidate AST rather than interpreter-specific nested bytecode; strict v2 comparisons remain strict, while legacy v1/no-source fallback is bounded and warns when it cannot prove identity;
• same-line candidates with list/dict/set comprehension bodies conservatively share the combined AST, so Python 3.11/3.12 and post-PEP-709 runtimes select the same recorded fingerprint instead of silently choosing different lambdas;
• a different-v1 physical open that is already present before the wait's terminal boundary can link to that proven chain, while a different-v1 open appearing after the terminal remains refused as an unrelated later keyless call;
• the append-only corpus and positive/refusal controls exercise both sides of that terminal boundary;
• the final d4cb87;�u����f49eacbd delta changes only the pre-existing heartbeat test from a fixed 200 ms sleep to an Event wait bounded at two seconds with cleanup in finally; worker runtime bytes are unchanged.

Independent diff review is clean. Owner qualification reports 211 focused pass / one skip plus Ruff; hosted qualification is terminal with 21/21 success across Python 3.12��y��y�3.15, integration, corpus, package, static analysis and target-branch gates. GitHub reports the head mergeable.

Scope: this clears General's exact-SHA merge of #260. It does not clear a 2.14.1 tag, wheel/sdist publication, or maintenance-branch head. Follow the recorded fix-only route from immutable 2.14.0, then return the exact maintenance head and installed-wheel 3.12-history-on-3.13 replay evidence for a separate Security go.

[rmcdaniel]

General accepts the four current Sonar maintainability findings at exact 49eacbdd20baa0ddb9f597d5456bffe00b4d5a74 for this replay repair. _predicate_node has one purpose, bounded AST selection, and exceeds the cognitive-complexity default by one point (16 versus 15); its explicit same-line/comprehension and missing-source branches preserve the reviewed compatibility rules. The three composite assertions each check command type and the required fingerprint together; splitting them would improve failure presentation but does not add missing behavioral coverage. None is a newly identified security or replay defect. Refactoring the cleared implementation is unnecessary for this urgent fix.

Security's exact-head review and all 21 current checks are green. General will merge only that SHA. The source branch is temporarily retained for the active fix-only 2.14.1 backport and must be removed after that maintenance handoff is complete. Maintenance-head review, installed-wheel acceptance and final tag/publication go remain separate.

[rmcdaniel]

General merged the Security-cleared exact head 49eacbdd20baa0ddb9f597d5456bffe00b4d5a74 as 16f217aa14628927beef1af9d22d3cc222dab63f. The merge tree equals the reviewed tree (698d8e6e1ddc69c77a13de3696c78e99a447072f), and the reviewed head is a merge parent. All 21 current-head checks were successful before merge.

The source branch remains temporarily for the active fix-only 2.14.1 backport. Next: prepare the maintenance draft from immutable 2.14.0 under #258's recorded route, qualify that head independently, then return its exact source and built-wheel replay evidence to Security. No tag/publication go is granted by this source merge.