Repository navigation
Supervisors inside the Horizon master (#7) - #8
Merged
Merged
Conversation
A unit's fork server forks, admits, signals and reaps the horizon:work workers of its supervisor loop. That handling moves into WorkerSet, unchanged, so another server can fork workers the same way. UnitServer keeps the loop and passes it every worker message, as before.
Some apps run Horizon's supervisors inside the master process instead of one process each. Their workers can now be forked as well: the master asks its template for each worker (a spawn with role: worker), and the template forks it through WorkerSet, with no fork server in between. The template TERMs the workers it forked when its master is gone, so none outlives it. On the master's side, Master::start() in inline mode (queue-prefork.horizon_inline) skips the check for the app's own AddSupervisor binding, applies the off switches itself and pumps on MasterSupervisorLooped. Master::workerProcess() hands out forked handles with a crash breaker per supervisor, and ForksWorkersFromTemplate lets the app's process pool use them. Horizon restarts a worker on the handle it has, so a stopped handle whose template is gone, or whose supervisor the breaker turned stock, is replaced with a new one, forked again or stock. The doctor reads horizon_inline as well.
A worker the template could not fork left its slot empty: Horizon restarted it on the same handle, which asked the template again on every restart. Its supervisor now turns stock (a UnitStock event), and the pool execs a worker in its place. The workbench runs Horizon's supervisors inside the master behind HORIZON_INLINE, and HorizonInlineTest drives real Horizon that way: the process tree, jobs, pause and continue, a killed worker, a failed fork (a process limit on the template) and horizon:terminate. CI runs it as its own Docker leg.
…mode Link and Master now count early worker exits through one Client\Breaker instead of two copies of the same rule. Compatibility::INLINE lists what ForksWorkersFromTemplate overrides, calls, reads and adds on Horizon's ProcessPool and WorkerProcess. Preflight checks it only in inline mode, so a change there never turns the package's own supervisors stock, and the daily check of new Horizon releases covers it through CompatibilityTest.
A handle kept the pause state of a pause sent before its worker was ready, and a continue sent after it never cleared it: the next run of the handle was admitted paused and took no jobs. Every pause and continue now updates it, whatever the handle's state.
- In inline mode the package no longer binds its own horizon command, which would replace the app's and its AddSupervisor; the package's command starts the master in regular mode explicitly. - Master::workerProcess() forks only for a master started inline. - Master reuses Wiring::horizonOff() instead of a copy of it. - HorizonInlineParityTest runs pause and continue, terminate and autoscaling on stock and prefork stacks that both run their supervisors inside the master, and compares them. - TemplateWorkersTest checks liveness by identity, so its shutdown test can't pass vacuously on macOS.
…k the pool after start() HorizonInlineParityTest now requires every worker of the prefork stack to be the template's child and every stock one the master's, and resumes both stacks even when a paused check fails. The README and the workbench create the pool with ForksWorkersFromTemplate only once Master::start() succeeded, so a Horizon that changed what the trait relies on never loads it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges #7 by @volodymyr-aloware, with follow-up commits on top. Their fork belongs to an organization, which doesn't accept maintainer pushes, so the branch lives here. Their three commits are unchanged.
Some apps run Horizon's supervisors inside the master process instead of one process each. With this, the template forks those supervisors' workers directly, on the master's request (
QUEUE_PREFORK_HORIZON_INLINE=true,Master::start()in the app's ownhorizoncommand,ForksWorkersFromTemplateon its process pool). Apps that don't opt in run exactly as before. #7 describes the design.Follow-up commits
LinkandMastercount early worker exits throughClient\Breakerinstead of two copies of the same rule.Compatibility::INLINElists what the trait overrides, calls, reads and adds on Horizon'sProcessPoolandWorkerProcess. Preflight checks it only in inline mode, so a Horizon change there never turns other apps stock.CompatibilityTestchecks the list against the trait, and the daily new-Horizon run executes it.horizoncommand stays the app's. In inline mode the package no longer binds its own command, which replaced the app's command and itsAddSupervisor.workerProcess()forks only for a master started inline.HorizonInlineParityTestruns pause and continue, terminate and autoscaling up and down on the stock and the prefork stack, both with supervisors inside the master, and compares them. The README and the workbench create the trait's pool only oncestart()succeeded.Tested
HORIZON_INLINE=true:HorizonInlineTestandHorizonInlineParityTest, 8 tests pass.