Skip to content

Supervisors inside the Horizon master (#7) - #8

Merged
tevkert merged 7 commits into
mainfrom
feat/inline-supervisors
Nov 20, 2026
Merged

tevkert merged 7 commits into
mainfrom
feat/inline-supervisors

Conversation

@tevkert

@tevkert tevkert commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Merges #7 by @volodymyr-aloware, with follow-up commits on top. Their fork belongs to an organization, which doesn't accept maintainer pushes, so the branch lives here. Their three commits are unchanged.

Some apps run Horizon's supervisors inside the master process instead of one process each. With this, the template forks those supervisors' workers directly, on the master's request (QUEUE_PREFORK_HORIZON_INLINE=true, Master::start() in the app's own horizon command, ForksWorkersFromTemplate on its process pool). Apps that don't opt in run exactly as before. #7 describes the design.

Follow-up commits

  • One crash breaker. Link and Master count early worker exits through Client\Breaker instead of two copies of the same rule.
  • Inline compatibility check. Compatibility::INLINE lists what the trait overrides, calls, reads and adds on Horizon's ProcessPool and WorkerProcess. Preflight checks it only in inline mode, so a Horizon change there never turns other apps stock. CompatibilityTest checks the list against the trait, and the daily new-Horizon run executes it.
  • Pause state across restarts (all Horizon setups). A worker paused before its first loop and continued after it was admitted paused on its next restart, and took no jobs until the next pause and continue. Every pause and continue now updates the state a restart starts in.
  • The app's horizon command stays the app's. In inline mode the package no longer binds its own command, which replaced the app's command and its AddSupervisor. workerProcess() forks only for a master started inline.
  • Tests against stock. HorizonInlineParityTest runs pause and continue, terminate and autoscaling up and down on the stock and the prefork stack, both with supervisors inside the master, and compares them. The README and the workbench create the trait's pool only once start() succeeded.

Tested

volodymyr-aloware and others added 7 commits November 19, 2026 18:44
A unit's fork server forks, admits, signals and reaps the horizon:work
workers of its supervisor loop. That handling moves into WorkerSet,
unchanged, so another server can fork workers the same way. UnitServer
keeps the loop and passes it every worker message, as before.
Some apps run Horizon's supervisors inside the master process instead of
one process each. Their workers can now be forked as well: the master asks
its template for each worker (a spawn with role: worker), and the template
forks it through WorkerSet, with no fork server in between. The template
TERMs the workers it forked when its master is gone, so none outlives it.

On the master's side, Master::start() in inline mode
(queue-prefork.horizon_inline) skips the check for the app's own
AddSupervisor binding, applies the off switches itself and pumps on
MasterSupervisorLooped. Master::workerProcess() hands out forked handles
with a crash breaker per supervisor, and ForksWorkersFromTemplate lets the
app's process pool use them. Horizon restarts a worker on the handle it
has, so a stopped handle whose template is gone, or whose supervisor the
breaker turned stock, is replaced with a new one, forked again or stock.
The doctor reads horizon_inline as well.
A worker the template could not fork left its slot empty: Horizon
restarted it on the same handle, which asked the template again on every
restart. Its supervisor now turns stock (a UnitStock event), and the pool
execs a worker in its place.

The workbench runs Horizon's supervisors inside the master behind
HORIZON_INLINE, and HorizonInlineTest drives real Horizon that way: the
process tree, jobs, pause and continue, a killed worker, a failed fork
(a process limit on the template) and horizon:terminate. CI runs it as
its own Docker leg.
…mode

Link and Master now count early worker exits through one Client\Breaker
instead of two copies of the same rule.

Compatibility::INLINE lists what ForksWorkersFromTemplate overrides,
calls, reads and adds on Horizon's ProcessPool and WorkerProcess.
Preflight checks it only in inline mode, so a change there never turns
the package's own supervisors stock, and the daily check of new Horizon
releases covers it through CompatibilityTest.
A handle kept the pause state of a pause sent before its worker was
ready, and a continue sent after it never cleared it: the next run of
the handle was admitted paused and took no jobs. Every pause and
continue now updates it, whatever the handle's state.
- In inline mode the package no longer binds its own horizon command,
  which would replace the app's and its AddSupervisor; the package's
  command starts the master in regular mode explicitly.
- Master::workerProcess() forks only for a master started inline.
- Master reuses Wiring::horizonOff() instead of a copy of it.
- HorizonInlineParityTest runs pause and continue, terminate and
  autoscaling on stock and prefork stacks that both run their
  supervisors inside the master, and compares them.
- TemplateWorkersTest checks liveness by identity, so its shutdown test
  can't pass vacuously on macOS.
…k the pool after start()

HorizonInlineParityTest now requires every worker of the prefork stack to be the template's child and every stock one the master's, and resumes both stacks even when a paused check fails. The README and the workbench create the pool with ForksWorkersFromTemplate only once Master::start() succeeded, so a Horizon that changed what the trait relies on never loads it.
@tevkert
tevkert merged commit 0ac36d1 into main Nov 20, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants