Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ jobs:
- { name: Horizon · stopasgroup off, horizon: 'true', host: 'true', stopasgroup: 'false', fast: 'true', redis: phpredis, compose: '', tests: tests/Integration, experimental: false }
- { name: Horizon · fast_termination off, horizon: 'true', host: 'true', stopasgroup: 'true', fast: 'false', redis: phpredis, compose: '', tests: tests/Integration, experimental: false }
- { name: Horizon · master spawns its template, horizon: 'true', host: 'false', stopasgroup: 'true', fast: 'true', redis: phpredis, compose: '', tests: 'tests/Integration/HorizonPreforkTest.php tests/Integration/LifecycleParityTest.php', experimental: false }
- { name: Horizon · supervisors inside the master, horizon: 'true', host: 'true', stopasgroup: 'true', fast: 'true', inline: 'true', redis: phpredis, compose: '', tests: tests/Integration/HorizonInlineTest.php, experimental: false }
- { name: Horizon · auto_prepend_file, horizon: 'true', host: 'true', stopasgroup: 'true', fast: 'true', redis: phpredis, compose: '-f compose.prepend.yml', tests: tests/Integration/StockParityTest.php, experimental: false }
- { name: queue:work, horizon: 'false', host: 'true', stopasgroup: 'true', fast: 'true', redis: phpredis, compose: '', tests: 'tests/Integration/QueueWorkParityTest.php tests/Integration/QueueWorkStockParityTest.php tests/Integration/QueueWorkLifecycleTest.php', experimental: false }
- { name: queue:work · stopasgroup off, horizon: 'false', host: 'true', stopasgroup: 'false', fast: 'true', redis: phpredis, compose: '', tests: 'tests/Integration/QueueWorkParityTest.php tests/Integration/QueueWorkStockParityTest.php tests/Integration/QueueWorkLifecycleTest.php', experimental: false }
Expand All @@ -163,6 +164,7 @@ jobs:
HORIZON_HOST: ${{ matrix.host }}
SUPERVISOR_STOPASGROUP: ${{ matrix.stopasgroup }}
HORIZON_FAST_TERMINATION: ${{ matrix.fast }}
HORIZON_INLINE: ${{ matrix.inline || 'false' }}
REDIS_CLIENT: ${{ matrix.redis }}
steps:
- uses: actions/checkout@v4
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

All notable changes to this project are documented here. The project uses [Semantic Versioning](https://semver.org/).

## [Unreleased]

- **Supervisors inside the master:** an app that runs Horizon's supervisors in the master process can have the template fork their workers directly (`horizon_inline`, `Master::start()`, `ForksWorkersFromTemplate`). The template's worker handling is shared with the fork servers (`WorkerSet`), and the template TERMs the workers it forked when its master is gone. A worker the template cannot fork turns its supervisor stock, and an exec'd worker takes its place.

## [1.1.2] - 2026-11-20

- **Stack traces in Flare, Sentry and Nightwatch:** a forked worker's stack trace ends in the launcher the template was started from, not in `artisan`. The launcher was a copy outside `vendor/`, so error trackers counted it as the app's own code. Flare drops every frame above the first frame of the app's code, so for a job failing in vendor code (a package's job, for example) only the launcher was left. Sentry marked it as the app's code, and Nightwatch placed such exceptions there. Templates now start through a link to the package's own `bin/template.php`, so traces name it in `vendor/`, and error trackers see the trace as in stock. Reported by Funsho Arashi (#5).
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,15 @@ Each `queue:work` process hands itself to a shared template before Laravel boots
- **Horizon works as usual,** because supervisor and worker code is Horizon's own: the dashboard, `horizon:terminate`, `pause`, `continue`, `purge`, autoscaling, and supervisord's `stopasgroup` and `killasgroup`.
- **Fallbacks.** A failed check runs the stock master in the same pid. If the template dies on Linux, Horizon exits and your process manager restarts it. A supervisor whose workers keep dying before their first loop is replaced by a stock one; a supervisor and its workers are always all forked or all stock. Each fallback logs a `queue-prefork:` error line and fires an event.

### Supervisors inside the master

Some apps run Horizon's supervisors inside the master instead of one process each. Their workers can still be forked: the template forks each worker itself, on the master's request, with no fork server in between.

- **Turn it on** with `QUEUE_PREFORK_HORIZON_INLINE=true` (config `horizon_inline`), which the doctor reads too.
- **In the app's own `horizon` command,** call `app(Tevkert\QueuePrefork\Master::class)->start()` before the provisioning plan deploys, when the container has that class bound (the package binds it only for `artisan horizon`). It returns false and changes nothing when the package is switched off or a check fails.
- **Give each in-process supervisor a process pool that uses `Tevkert\QueuePrefork\Horizon\ForksWorkersFromTemplate`.** Its workers are forked while the master runs this way and exec'd as stock otherwise. A worker the template cannot fork turns its supervisor stock (a `UnitStock` event). A stopped worker whose template is gone, or whose supervisor turned stock, gets a new handle, forked again or stock, instead of being restarted on the old one.
- **Don't run `horizon:purge` in this mode.** It looks for workers among the children of supervisor processes, and these workers are the template's children.

### Fork safety

Before forking, a template audits itself: sockets, epoll, eventfd or extra threads that extensions open at startup mean stock workers. ddtrace, Elastic APM and gRPC with fork support (`grpc.enable_fork_support=1`, which rebuilds its threads and poller in each fork) are allowed; workers run stock with New Relic, gRPC without fork support, datadog-profiling and Blackfire.
Expand Down
5 changes: 5 additions & 0 deletions config/queue-prefork.php
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@
'queue_work' => env('QUEUE_PREFORK_QUEUE_WORK', true),
'horizon' => env('QUEUE_PREFORK_HORIZON', true),

// The app runs Horizon's supervisors inside the master and asks the template only for their workers
// (Master::start(inline: true), ForksWorkersFromTemplate). Its own AddSupervisor binding is then expected;
// the doctor checks this mode instead of the forked supervisors.
'horizon_inline' => env('QUEUE_PREFORK_HORIZON_INLINE', false),

// The templates' opcache. Its shared memory holds the compiled code of the whole app.
'opcache' => [
'memory_consumption' => env('QUEUE_PREFORK_OPCACHE_MB', 192),
Expand Down
4 changes: 4 additions & 0 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,9 @@ HP_INTEGRATION=1 vendor/bin/phpunit --testsuite Integration # Horizon's tests
HORIZON_HOST=false docker compose -f docker/compose.yml up -d --force-recreate app-prefork # the master spawns its template (fallback mode)
HP_INTEGRATION=1 vendor/bin/phpunit tests/Integration/HorizonPreforkTest.php tests/Integration/LifecycleParityTest.php
docker compose -f docker/compose.yml up -d --force-recreate app-prefork # back to host mode
HORIZON_INLINE=true docker compose -f docker/compose.yml up -d --force-recreate app-prefork # supervisors inside the master
HP_INTEGRATION=1 vendor/bin/phpunit tests/Integration/HorizonInlineTest.php # skips unless the stack runs inline
docker compose -f docker/compose.yml up -d --force-recreate app-prefork # back to a process per supervisor
HORIZON=false QUEUE_WORK=true docker compose -f docker/compose.yml up -d --force-recreate app-stock app-prefork
HP_INTEGRATION=1 vendor/bin/phpunit tests/Integration/QueueWorkParityTest.php
```
Expand Down Expand Up @@ -136,6 +139,7 @@ Host environment, read when a container is created (`up -d --force-recreate`):
| `QUEUE_WORK_STOPWAIT` | `10` | their supervisord `stopwaitsecs` |
| `SUPERVISOR_STOPASGROUP` | `true` | supervisord `stopasgroup` (`killasgroup` is always `true`) |
| `HORIZON_FAST_TERMINATION` | `true` | `horizon.fast_termination` |
| `HORIZON_INLINE` | `false` | supervisors inside the master (the workbench's `App\Horizon` classes), the template forking their workers |
| `CONFIG_CACHE` | `true` | `optimize` + `filament:optimize` at start; `false` runs uncached |
| `NIGHTWATCH_ENABLED` / `PULSE_ENABLED` | `true` / `true` | switch the two monitoring packages off, e.g. to measure their cost |
| `SOAK_DISPATCH` / `SOAK_RATE` / `SOAK_DURATION` / `SOAK_SEED` | `false` / `420` / `60` / `1` | the `soak-dispatch` program (jobs/min, minutes, seed) |
Expand Down
4 changes: 4 additions & 0 deletions docker/compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
# (default true), or spawning its template as a child
# SUPERVISOR_STOPASGROUP=true|false supervisord group mode (default true)
# HORIZON_FAST_TERMINATION=true|false horizon.fast_termination (default true)
# HORIZON_INLINE=true|false supervisors inside the master, the template forking their
# workers (default false)
# CONFIG_CACHE=true|false config:cache at start (default true)
# SOAK_DISPATCH, SOAK_MIX, SOAK_RATE, SOAK_DURATION, SOAK_SEED, SOAK_SAMPLE, SOAK_SAMPLE_INTERVAL the soak
name: queue-prefork
Expand Down Expand Up @@ -68,6 +70,8 @@ x-app-env: &app-env
HORIZON_HOST: ${HORIZON_HOST:-true}
SUPERVISOR_STOPASGROUP: ${SUPERVISOR_STOPASGROUP:-true}
HORIZON_FAST_TERMINATION: ${HORIZON_FAST_TERMINATION:-true}
HORIZON_INLINE: ${HORIZON_INLINE:-false}
QUEUE_PREFORK_HORIZON_INLINE: ${HORIZON_INLINE:-false}
CONFIG_CACHE: ${CONFIG_CACHE:-true}
SOAK_DISPATCH: ${SOAK_DISPATCH:-false}
SOAK_MIX: ${SOAK_MIX:-heavy}
Expand Down
7 changes: 7 additions & 0 deletions src/Client/ForkedProcess.php
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ public function start(?callable $callback = null, array $env = []): void
$this->broker->request("spawn:{$this->id}", [
't' => 'spawn',
'id' => $this->id,
'role' => $this->role,
'cmd' => $this->line,
'cwd' => $this->directory,
'env' => getenv(),
Expand Down Expand Up @@ -216,6 +217,12 @@ public function ready(): void
}
}

/** Whether the server this handle forks through is gone: starting it again can only fail. */
public function serverGone(): bool
{
return $this->broker->lost;
}

/** @internal */
public function serverLost(): void
{
Expand Down
3 changes: 2 additions & 1 deletion src/Console/DoctorCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ public function handle(): int
$horizon = class_exists(MasterSupervisor::class);
$release = realpath(base_path()) ?: base_path();
$pidOne = PHP_OS_FAMILY === 'Linux' ? str_replace("\0", ' ', (string) @file_get_contents('/proc/1/cmdline')) : '';
$reasons = $horizon ? [...Switches::off($release, 'horizon'), ...Preflight::check($this->laravel), ...self::pidOne($pidOne, 'horizon')] : [];
$inline = (bool) $this->laravel['config']->get('queue-prefork.horizon_inline', false);
$reasons = $horizon ? [...Switches::off($release, 'horizon'), ...Preflight::check($this->laravel, inline: $inline), ...self::pidOne($pidOne, 'horizon')] : [];
$queueWork = self::queueWork($release, self::pidOne($pidOne, 'queue:work'));
$warnings = [...self::hostWarnings($horizon), ...$this->earlyMismatches($release)];

Expand Down
60 changes: 60 additions & 0 deletions src/Horizon/ForksWorkersFromTemplate.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<?php

namespace Tevkert\QueuePrefork\Horizon;

use Laravel\Horizon\WorkerProcess;
use Tevkert\QueuePrefork\Client\ForkedProcess;
use Tevkert\QueuePrefork\Master;

/**
* For the ProcessPool of a supervisor that runs inside the Horizon master: its workers are forked by the
* template once the master started in inline mode (Master::start(), queue-prefork.horizon_inline), and
* exec'd as stock otherwise.
*/
trait ForksWorkersFromTemplate
{
protected function createProcess()
{
$process = $this->preforkMaster()?->workerProcess($this->options);

return $process === null ? parent::createProcess() : new WorkerProcess($process);
}

public function monitor()
{
$this->replaceRetiredWorkers();

parent::monitor();
}

/**
* Horizon restarts a stopped worker on the handle it has. A forked handle whose template is gone, or whose
* supervisor turned stock (the crash breaker, or a worker the template could not fork), gets a new one
* instead: forked again, or stock.
*/
protected function replaceRetiredWorkers(): void
{
$master = $this->preforkMaster();

foreach ($this->processes as $index => $worker) {
$process = $worker->process;

if (! $process instanceof ForkedProcess || $process->isRunning()) {
continue;
}

if ($process->serverGone() || $master?->retires($process, $this->options->name)) {
$this->processes[$index] = $this->createProcess()->handleOutputUsing(function ($type, $line) {
call_user_func($this->output, $type, $line);
});
}
}
}

private function preforkMaster(): ?Master
{
$app = app();

return $app->bound(Master::class) ? $app->make(Master::class) : null;
}
}
121 changes: 119 additions & 2 deletions src/Master.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

use Illuminate\Contracts\Foundation\Application;
use Illuminate\Support\Facades\Redis;
use Laravel\Horizon\Events\MasterSupervisorLooped;
use Laravel\Horizon\MasterSupervisor;
use Laravel\Horizon\SupervisorOptions;
use Tevkert\QueuePrefork\Client\Broker;
Expand Down Expand Up @@ -75,20 +76,40 @@ final class Master
/** @var array<string, ForkedProcess> supervisor name => current unit handle */
private array $units = [];

/** @var array<string, list<float>> supervisor name => recent exits of its workers before their first loop */
private array $earlyWorkerExits = [];

public function __construct(private Application $app) {}

public function start(?callable $warn = null): bool
/**
* @param bool|null $inline the app runs its supervisors in this process and asks for their workers through
* workerProcess(), so its own AddSupervisor binding is expected, not a conflict;
* null reads queue-prefork.horizon_inline, which the doctor reads too
*/
public function start(?callable $warn = null, ?bool $inline = null): bool
{
Runtime::preload();
$inline ??= (bool) $this->app['config']->get('queue-prefork.horizon_inline', false);

if ($inline && ($off = $this->inlineOff()) !== null) {
$this->reason = $off;

return false;
}

try {
$reasons = Preflight::check($this->app);
$reasons = Preflight::check($this->app, inline: $inline);

if ($reasons === [] && $this->startTemplate()) {
$this->active = true;
$this->mode = 'prefork';
Report::event('PreforkStarted', ['template' => $this->template['pid'] ?? null], 'info');

if ($inline) {
// The app's own horizon command runs the loop: nothing else registers this.
$this->app['events']->listen(MasterSupervisorLooped::class, fn () => $this->tick());
}

return true;
}

Expand All @@ -113,6 +134,21 @@ public function start(?callable $warn = null): bool
return false;
}

/**
* Why an inline start stays stock: Wiring checks the same switches before it binds the package's own
* command, which an app running its supervisors in the master replaces with its own.
*/
private function inlineOff(): ?string
{
$off = Support\Switches::off(base_path(), 'horizon');

if ($off !== []) {
return implode('; ', $off);
}

return Settings::enabled($this->app, 'horizon') ? null : 'switched off ('.Settings::switchSource($this->app, 'horizon').')';
}

/** Called on every MasterSupervisorLooped (and so also while paused). */
public function tick(): void
{
Expand Down Expand Up @@ -158,6 +194,87 @@ public function unitProcess(SupervisorOptions $options): ?ForkedProcess
return $this->units[$options->name] = $unit;
}

/**
* A worker forked by the template for a supervisor that runs in this process, or null for a stock one:
* when nothing forks, for a command line the template does not run, once the supervisor's workers kept
* exiting before their first loop, or once the template could not fork one of them. A handle once forked
* stays forked when its worker restarts.
*/
public function workerProcess(SupervisorOptions $options): ?ForkedProcess
{
if (! $this->canForkUnits() || isset($this->stockNames[$options->name])) {
return null;
}

$line = $options->toWorkerCommand();

try {
CommandLine::parse($line, ['horizon:work']);
} catch (Throwable $e) {
$this->stockNames[$options->name] = $e->getMessage();

return null;
}

$worker = new ForkedProcess($this->broker, 'worker', $line, $options->directory ?? base_path(), ! config('horizon.fast_termination'));
$worker->onExit = function (ForkedProcess $process, int $status, bool $wasReady) use ($options): void {
$this->workerExited($options->name, $process, $status, $wasReady);
};
$worker->onSpawnFailed = function (ForkedProcess $process, string $reason) use ($options): void {
$this->workerSpawnFailed($options->name, $process, $reason);
};

return $worker;
}

/**
* Whether a stopped forked worker of $supervisor needs a new handle: Horizon restarts a worker on the
* handle it has, which still forks through a template that is gone, or for a supervisor now stock.
*/
public function retires(ForkedProcess $worker, string $supervisor): bool
{
return $worker->serverGone() || isset($this->stockNames[$supervisor]);
}

/**
* A worker the template could not fork turns its supervisor stock: the pool replaces the handle with an
* exec'd worker instead of asking the template again on every restart. A lost template needs nothing here:
* every handle retires and nothing forks any more.
*/
private function workerSpawnFailed(string $supervisor, ForkedProcess $process, string $reason): void
{
if ($process->serverGone() || isset($this->stockNames[$supervisor])) {
return;
}

$this->stockNames[$supervisor] = 'worker spawn failed: '.$reason;
Report::event('UnitStock', ['supervisor' => $supervisor, 'reason' => $this->stockNames[$supervisor]]);
}

/** The unit breaker of Link::workerExited(), kept here per supervisor, as no loop of its own runs it. */
private function workerExited(string $supervisor, ForkedProcess $process, int $status, bool $wasReady): void
{
if ($wasReady || $status === 0 || $process->parentSignalled || in_array($status, [137, 75, 72], true)) {
return;
}

$now = microtime(true);

if ($now - $process->spawnedAt > 120) {
return;
}

$window = (float) Runtime::config('breaker_window', 300);
$exits = array_values(array_filter($this->earlyWorkerExits[$supervisor] ?? [], fn (float $at) => $now - $at < $window));
$exits[] = $now;
$this->earlyWorkerExits[$supervisor] = $exits;

if (count($exits) >= (int) Runtime::config('breaker_exits', 5)) {
$this->stockNames[$supervisor] = 'crash breaker';
Report::event('BreakerTripped', ['scope' => 'supervisor', 'supervisor' => $supervisor, 'status' => $status]);
}
}

public function statusDir(): string
{
return $this->statusDir;
Expand Down
Loading