Repository navigation
CxODEV-2161: keep the engine polling after Conductor errors (4.4.2) - #233
Draft
milansvitlica-codaxy wants to merge 4 commits into
Draft
milansvitlica-codaxy wants to merge 4 commits into
milansvitlica-codaxy wants to merge 4 commits into
Conversation
…t is not a shutdown (CxODEV-2000) WorkflowEngineBackgroundService ran the deployment and the execution manager once. Any error, or an HTTP timeout mistaken for a shutdown, ended polling for good while the host stayed up (and, for a timeout, stayed Healthy). RunAsync is now a supervisor loop: deploy, mark running, run the execution manager; on any error that is not our own shutdown, mark not running, log an error, back off (1 s doubling to 30 s, 500 ms jitter) and start again. The backoff resets after a successful deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A real engine (DI wiring, hosted service, beta execution manager, HTTP client) against a fake Conductor that answers 503, hangs (request timeout) or loses the task-def registration race, then recovers. Each test checks that the engine backs off, reports not running, and completes a task queued after recovery; one also checks that a task in flight during the outage still completes. All four fail on the old loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shutdown cancel a hung deploy Review findings on the supervisor loop: - Health and the backoff reset followed a successful deploy, not a successful poll. With Conductor's metadata up and its task queues down (e.g. the queue backend), the engine reported Healthy most of the time while polling nothing, and redeployed about once a second on a 4xx. TaskQueuePollingService now marks the engine running after a successful read; the background service no longer marks it after deploy and restarts the backoff only after a run that polled. Unhealthy is still set only by the background service. - Deploy takes no token, so a hung deploy blocked StopAsync. It now runs under WaitAsync(stoppingToken). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes CxODEV-2161 and CxODEV-2000 (YouTrack). A ConductorSharp engine no longer stops polling for good after a Conductor error.
Root cause:
WorkflowEngineBackgroundService.RunAsyncran deploy + execution manager once. Any exception ended polling while the host stayed up. An HTTP timeout (TaskCanceledException) was also taken for a shutdown: the engine loggedINF Stopping ConductorSharp background serviceand stayed Healthy.Fix:
RunAsyncis a supervisor loop. It deploys and runs the execution manager. On any error that is not our own shutdown, it marks the engine not running, logs an Error, backs off (1 s doubling to 30 s, ≤ 500 ms jitter) and starts again.TaskQueuePollingServicemarks the engine running after a successful queue read, so/healthis Healthy only while the engine actually reads its queues. Only a run that polled restarts the backoff. A shutdown cancels a hung deploy.TaskQueuePollingService's bounded retry (#215) stays as the inner, fast layer.Tests
Unit/WorkflowEngineBackgroundServiceTests(11): retry of deploy errors, restart after a connection error, timeout ≠ shutdown, health while backing off, capped backoff, backoff reset only after a successful poll, deploy alone does not mark the engine running, stop during backoff, stop while a deploy hangs, stop while running.Unit/TaskQueuePollingServiceTests(+2): a successful read marks the engine running; failed reads do not.Integration/EngineRecoveryTests(5): a real engine (DI, hosted service, beta execution manager, HTTP client) against a fake Conductor. Covers a 503 outage, a hang, the registration race, a task in flight during an outage, and task queues failing while metadata works (stays Unhealthy). All fail on the old loop.DOTNET_ROLL_FORWARD=Major, LF checkout).Reproduced before the fix (2026-10-19), on a local cxo-composer stack with SOE/ROE/locking on 3.10.0: after
docker pauseon Conductor, all three engines stopped 8 min 36 s later and stayedhealthy. Afterdocker stop+start, they logged FTL, turned 503 and never reconnected. Only a restart brought them back.Behaviour change to note: the old
FTL Workflow Engine Background Service encountered an (API) error(s)is gone. Each failed attempt now logs at Error:ConductorSharp engine stopped polling (attempt N); retrying in …. Alerts that matched the old FTL text need updating.Version: 4.4.2 (all five packages). The v3 backport (3.10.1) follows in a separate PR; CxO engines pin 3.10.x.
🤖 Generated with Claude Code