[Feature]: Add virtual_parent_id to plugin metadata and preserve virtual-context span hierarchy #890

What would you like?

Add optional virtual_parent_id to plugin operation metadata and update both Execution and Invocation OTel plugins to parent operations under their direct virtual context.

Virtual contexts already produce OTel spans, but have no durable CONTEXT checkpoint. Their operations use the nearest checkpointed ancestor as parent, so the virtual span and its operations appear as siblings. This operation parentage matches durable execution history. The proposed change deliberately makes tracing include the logical virtual-context hierarchy while leaving the history hierarchy unchanged.

Metadata contract

• virtual_parent_id is the stable operation ID of the direct owning parent context, only when that parent is virtual. Otherwise leave it None. Use the same ID representation as existing plugin metadata.
• It is not the nearest virtual ancestor through an intervening regular context. For virtual A -> regular B -> step C, B identifies A as its virtual parent; C leaves the new field absent and keeps its existing parent B. For virtual A -> virtual B -> step C, B identifies A and C identifies B.
• Preserve parent_id. OTel selects virtual_parent_id when present, otherwise parent_id. Step attempt spans remain children of their step operation.
• This is instrumentation metadata only: no virtual START/SUCCEED/FAIL checkpoints, additional durable operations or checkpoint requests, or serialized payload wrappers. Operation IDs, wire ParentId values, checkpoint payloads, validation, results, and replay semantics must remain unchanged.

Possible implementation

• Register operation-to-virtual-parent ownership when the core allocates/traverses the stable operation ID, before replay inspection or a cached-result early return. Do not rerun completed user code or fabricate lifecycle events to learn ancestry.
• Keep the registry invocation-local and safe for concurrent child contexts. Supply known ownership to applicable lifecycle hooks and checkpoint-derived replay/update/snapshot metadata, including waits without a user-function body.
• Completion snapshots can arrive before traversal establishes the virtual parent. Defer parent resolution where possible and prefer subsequently learned ownership over a raw backend parent. Retain completed parent SpanContexts until invocation cleanup for late children.
• If a virtual scope is never traversed and no source supplies its ancestry, leave the field absent and retain the documented existing-parent fallback. Flattened history and opaque IDs cannot reconstruct an unrecorded relationship; do not add persistence or guess.

Both OTel views must consume the field:

• Execution: children emitted before suspension and after resume must reference the same logical virtual parent identity as the eventual exported virtual span. Avoid dangling references to discarded placeholders.

• Invocation: resolve the actual virtual segment from the current invocation, including completion-only spans. Different invocations need distinct segment identities; never substitute an earlier invocation's segment.

• Use the idiomatic Python spelling virtual_parent_id for the shared virtualParentId contract. Add an optional field defaulting to None without breaking existing metadata construction.

• Cover operation/user-function metadata and checkpoint/replay completion conversions; populating only user-function events misses waits and asynchronous updates.

• Align virtual Execution placeholder identity with the final exported parent across invocations. Current invocation-specific placeholders would otherwise leave early children pointing at discarded spans.

• Coordinate the separate premature REPLAY-to-NEW transition at virtual entry, which can suppress the resumed wait update hook. Parent metadata does not restore an event that is never delivered; resolve or explicitly link that dependency so the public wait/resume regression passes.

Reproduction and acceptance criteria

Use the public API to run otel-virtual-context as a virtual child containing step otel-virtual-before, a five-second durable wait otel-virtual-wait, and step otel-virtual-after. Force suspension/resume and inspect raw exporter records in both views.

• Direct virtual-parent metadata and raw exported parent IDs are correct on initial execution and resume; attempts remain under their operations.
• Cover nested virtual contexts, a regular context inside a virtual context, a step inside that regular context, synchronous failure, and concurrent sibling contexts/FLAT branches.
• Cover completion without local START, known-parent updates arriving before traversal or after the parent body returns, and unknown-parent fallback separately.
• Verify zero added virtual checkpoints, unchanged operation IDs/wire parents/payloads, stable results and replayed failures, and no repeated completed-step side effects.
• Preserve existing metadata construction and plugins without the field. Document the new field, view-specific rules, fallback limits, and the deliberate difference from execution-history nesting.

Coordinate any shared conformance requirement separately; implement focused core/OTel tests and public-handler regressions in this SDK.

Scope and compatibility

Original virtual start-time recovery and general exactly-once export across later replays remain separate lifecycle work. Consistent parent identities are required here; adding this field must not be described as solving timestamp recovery or export deduplication.

Is this a breaking change? Make the metadata extension additive, preserving existing construction paths and fallback. Virtual-child OTel parentage intentionally changes for trace consumers; document it in release notes and decide the appropriate compatibility/versioning treatment.

Does this require an RFC? Yes: align this tracing hierarchy contract across JavaScript, Python, and Java before implementing the behavior change.

Evidence and related work

Source inspected at commit 44b9f4d1c862b50041423e19fe6a250bbf7bd72f: child context construction preserves the ancestor durable parent; Execution parent resolution uses it. Virtual placeholder creation currently uses invocation-specific identity. Implement against current main and reuse newer completion handling.

• #854 / PR #869 cover visible replay completions and retained parent contexts, but do not define this field.
• JS #997 tracks the separate completed-virtual replay export problem.

Companion SDK issues

• JavaScript #1136
• Java #855