Skip to content

test: cover OTel replay and status mappings - #758

Open
zhongkechen wants to merge 41 commits into
mainfrom
test/otel-conformance-feasibility
Open

zhongkechen wants to merge 41 commits into
mainfrom
test/otel-conformance-feasibility

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Add Python handlers and SAM mappings for OTel cases 21–26: completed-step replay, active context in user functions and callbacks, invocation retry status, callback failure without error details, and external callback completion followed by two later replays. Handlers use public durable APIs and ordinary implicit span parenting.

Case 26 creates a root callback, saves its result in a durable observation step, and uses two callback barriers to force four invocations. The shared driver delivers callbacks after InvocationCompleted; raw S3 assertions require the first terminal export before the observation step, under the second Invocation in Invocation view, and reject duplicate exports on later replays. Case 25 reuses the callback-failure handler with an omitted payload and strict typed-empty-history-payload precondition.

The testing library reports callback success/failure/timeout through UpdatedOperationIds and consumes only versions delivered by successful checkpoints. Core terminal notifications track actual delivery per invocation, preserving first notifications, concurrent delivery and resets. Exactly empty callback errors become absent in SDK-facing state while detailed history retains Error.Payload={} and Truncated=false, including typed round trips. Present error fields, metadata-only history and checkpoint tokens/watermarks retain their behavior.

A sibling checkpoint can deliver a terminal operation before replay creates its parent span. Invocation view now retains the real completion until that actual parent span is registered, then exports the terminal segment before returning control to the branch. It does not invent ancestors or alter SDK notification accounting. Unresolved parents are reported after normal cleanup/flush, and pending state is cleared on reuse. A deterministic public two-callback regression verifies both original nonterminal and terminal segments, their real parents, the observation boundary and two later replays.

The #756 prerequisite moves Start, handler/finally, existing output/error preparation, resource shutdown and End onto one worker; the caller owns executor shutdown. It removes the extra handler_context API while retaining failed-Start Context/token ownership. Registered-plugin map/parallel submissions receive fresh coordinator Context copies, including resumes, preserving sibling isolation and the no-plugin path. The required reused-worker executor unit test is included. OTel 1.1 requires redesigned core 2.1, released first.

The reusable workflow remains bdb4f1cd0f9252c1aaa978bb8b341b71f2b9d9dc. Runner/catalog default and fallback are 75987d46a915bc37409eed3ea9c3617a924c9756 from aws/aws-durable-execution-conformance-tests#131, including shared #120's 120-second log polling, accumulation and deduplication. The 26 requirements, handlers/templates, backend matrix, long-running scenarios, resource locks and failed-plus-uncovered gates remain intact. The shared self-test fixture stays on its already-validated reference 12e760c; SDK CI tests this PR's exact head.

Merge prerequisites: #756 and aws/aws-durable-execution-conformance-tests#131. Status-mapping prerequisite #752 is merged.

Coverage boundary: case 24 verifies RETRY to RETRYING/UNSET. Case 25 verifies FAILED without error details to UNSET and typed empty history payloads in both views. CANCELLED, TIMED_OUT and STOPPED without error details retain unit coverage; their cloud paths are not established. On preceding head 21be89e, OTel run 37860686614 verifies all four 26/26 reports and both 4/4 long-running reports with no failures or uncovered cases. Both S3 views' actual case-25/26 histories were inspected. Current head d59af28 is now fully green. OTel run 37871824929 resolved runner 75987d46a915bc37409eed3ea9c3617a924c9756 and passes all four 26/26 plus both 4/4 reports (112 cases), with zero failed/uncovered cases. Both S3 views' case-25/26 raw histories were verified. The matching strict reference test is updated and its 83-script-test suite passes.

Validation:

  • Full repository: 4,417 passed, 2 skipped, followed by the unit-only prerequisite port's 203 passing concurrency tests. Final core coverage: 1,820 passed, 98.25%, above the 98% gate.
  • Focused Invocation tests: 131 passed, including the unchanged strict 40-case nested suite, deterministic all-segments public regression, and status/error/nested-parent/unresolved/reuse controls.
  • Installed artifacts: 570 OTel tests, 12 current-pair controls, and all 40 strict nested controls pass using the actual built core, OTel and testing-library wheels. The separate released OTel 1.0/new-core lane passes 12 controls. Artifact hashes and all installed Python source bytes are verified; no editable source shadows these lanes. The testing wheel retains its existing 1.2.1 version label, so source/artifact hashes distinguish this PR's fixes from the published package.
  • Types (80 core, 33 OTel, 42 insight files), Hatch lint/format, workflow/reference checks and commit lint pass. No test assertion, case or gate was removed to make the repair pass.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 3, 2026 00:14 — with GitHub Actions Active
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 3, 2026 00:19 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen zhongkechen changed the title test(otel): cover replay and callback contexts test: add OTel conformance handlers 21–24 Oct 3, 2026
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 3, 2026 00:57 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen zhongkechen changed the title test: add OTel conformance handlers 21–24 test: add otel context and replay conformance Oct 3, 2026
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 3, 2026 02:08 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen zhongkechen changed the title test: add otel context and replay conformance test: cover OTel replay and user callback contexts Oct 3, 2026
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 3, 2026 02:26 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen changed the base branch from fix/otel-handler-context-428 to main October 3, 2026 02:37
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 3, 2026 02:38 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 8, 2026 01:19 — with GitHub Actions Active
@github-actions

This comment has been minimized.

Represent an exactly empty serialized callback error as absent, matching
CallbackDetails.from_dict without requiring a file-store round trip.
Preserve every present field, including empty strings and stack lists.

Compare real public handler failures across memory/file stores and both
OTel views, including caller error/status and replay behavior.

Pin the coordinated shared S1c revision with the strict empty-payload,
parent-occurrence and callback phase-gate assertions.
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 8, 2026 01:41 — with GitHub Actions Active
@github-actions

This comment has been minimized.

Match the observed service projection for a failed callback with no error
details when IncludeExecutionData is true: keep Payload {} and mark it
untruncated. SDK-facing state remains error=None, with an UNSET leaf and
the same caller failure. Other statuses, nonempty errors and metadata-only
history retain their existing representation.

Verify the raw public history shape, state/telemetry separation and direct
factory boundaries. The unchanged JS 19ee19f example suite passes all 148
tests without retries or assertion/selection changes.
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 8, 2026 02:09 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen

zhongkechen commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

@hln33 Follow-up to your #752 coverage comment: this PR now adds case 25 for the missing non-success/no-error-details mapping, alongside the existing case 24 RETRYING/UNSET coverage.

Case 25 uses the public callback-failure handler and sends failure without Error after InvocationCompleted. The shared requirement checks the typed empty raw error payload and requires the FAILED callback leaf to remain UNSET in both views. Returning OK for that branch would now fail conformance. Case 26 also exercises a root callback’s first completion, its second-invocation parent, and two subsequent replays without duplicate terminal exports.

On current commit cd81a3d, S1c cloud run 37716434728 reports 26/26 passed in all four backend/view suites, with zero failed or uncovered cases; both long-running suites pass 4/4. The unchanged JS examples CI also passes 148 tests on this head.

FAILED, CANCELLED, TIMED_OUT and STOPPED without error details remain covered by the two-view unit matrix. The new real-service case exercises FAILED; no cloud reachability claim is made for the other three statuses. The separate new typed-history round-trip review is being investigated as testing-API maintenance; it does not change this deployed handler coverage.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 8, 2026 02:45 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 8, 2026 21:13 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 8, 2026 23:40 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 9, 2026 01:52 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings found. Residual risk remains in timing-sensitive worker/context propagation and cloud OTel export ordering, which were reviewed statically.

Reviewed commit d59af28c25fc73d7423cab456a0a8fa90641509e. Workflow run

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — d59af28c Deployed Oct 9, 2026 by zhongkechen via ai-pr-review / Codex review / Generate Codex review #1465
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants