Repository navigation
CxODEV-2161: keep the engine polling after Conductor errors (3.10.1 backport) - #234
Draft
milansvitlica-codaxy wants to merge 4 commits into
Draft
milansvitlica-codaxy wants to merge 4 commits into
milansvitlica-codaxy wants to merge 4 commits into
Conversation
…t is not a shutdown (CxODEV-2000) WorkflowEngineBackgroundService ran the deployment and the execution manager once. Any error, or an HTTP timeout mistaken for a shutdown, ended polling for good while the host stayed up (and, for a timeout, stayed Healthy). RunAsync is now a supervisor loop: deploy, mark running, run the execution manager; on any error that is not our own shutdown, mark not running, log an error, back off (1 s doubling to 30 s, 500 ms jitter) and start again. The backoff resets after a successful deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A real engine (DI wiring, hosted service, beta execution manager, HTTP client) against a fake Conductor that answers 503, hangs (request timeout) or loses the task-def registration race, then recovers. Each test checks that the engine backs off, reports not running, and completes a task queued after recovery; one also checks that a task in flight during the outage still completes. All four fail on the old loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shutdown cancel a hung deploy Review findings on the supervisor loop: - Health and the backoff reset followed a successful deploy, not a successful poll. With Conductor's metadata up and its task queues down (e.g. the queue backend), the engine reported Healthy most of the time while polling nothing, and redeployed about once a second on a 4xx. TaskQueuePollingService now marks the engine running after a successful read; the background service no longer marks it after deploy and restarts the backoff only after a run that polled. Unhealthy is still set only by the background service. - Deploy takes no token, so a hung deploy blocked StopAsync. It now runs under WaitAsync(stoppingToken). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #233 to
v3(3.10.1). Fixes CxODEV-2161 and CxODEV-2000 (YouTrack) for the CxO engines, which pin ConductorSharp 3.10.x.The three code commits from #233 cherry-picked cleanly:
WorkflowEngineBackgroundService.csandTaskQueuePollingService.cswere identical onmasterandv3. With them:/healthis Healthy only while the engine actually reads its queues.See #233 for the root cause, design and before/after table.
Behaviour change to note: the old
FTL Workflow Engine Background Service encountered an (API) error(s)is gone. Each failed attempt now logs at Error:ConductorSharp engine stopped polling (attempt N); retrying in …. Alerts that matched the old FTL text need updating.Versions: Client, Engine, Patterns and KafkaCancellationNotifier go to 3.10.1. Toolkit stays 3.0.1-beta3.
Tests:
v3PRs get no CI, so this is the local run (LF checkout, net6.0 viaDOTNET_ROLL_FORWARD=Major): 88/88 passed, including the 13 new unit tests and the 5 end-to-end recovery tests.🤖 Generated with Claude Code