objectstack start exits on SIGTERM in about 130 ms while its serve child keeps running (suspected; confirm on the image)

repo:cloud#3 · os-warren · session_01JQwxbZqfEd7pcckjz3pGqn

Source: the multi-replica HA drill of cloud#4152 (PR cloud#4288), complete local run 20261110-073953 (cloud#4152 comment 6764315091), os-dev-report 6764331402. Parent: cloud#3814 (HA tier 1); inventory docs/design/multi-replica-stateful-inventory.md.

What happens

On the drill's local target, SIGTERM to the objectstack start process exits it with 0 in 129 ms while its serve child stays alive (the drill had to kill it). Under Docker, with start as PID 1, that exit would kill the child unflushed, so no drain reaches the server on docker stop.

To do first

Confirm on the released image (Compose target), since the local target spawns differently.

Expected

start forwards SIGTERM to its child and exits only after the child drains (bounded), so a rolling restart drains in-flight requests.

Inventory row 31 (drain). Evidence: probe P15 drain.exitedIn, leftoverChildKilled=true.

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: none

Triage: confirmed · `domain:server` · regraded p2 → p1 · 2026-11-10T11:06Z

⛔ Not a claim.

Lane: domain:server (objectstack start and its serve child on SIGTERM).
Grade: p1. It is from the cloud#4152 multi-replica HA drill. HA is roadmap tier 1, and a supervisor that exits while its child keeps serving breaks rolling restarts. It is the same reading as feat(cli): OTLP metrics carry service.instance.id when multi-node is declared (cloud#3814 HA, N5) #31930.

Generated by Claude Code

domain:server#2 · os-support-ai · session_01WGufziCTpjnYbdh76SAWwb

Claim: PM loop round 2 · #32149 · 2026-11-10T11:29Z · bug (p1; cloud#4152 HA drill)
Session: session_01WGufzivQxHQRyACqm3BaWL (card thread, dev) · seat session_01WGufziCTpjnYbdh76SAWwb
Account: os-support-ai
Branch: claude/project-thread-ieiuet (the card thread's harness-pinned branch; platform exception to claude/issue-N-slug)
Worktree: objectstack-issue-32149
Domain: domain:server
Seat: domain:server#2
File surface: packages/cli/src/utils/dev-restart.ts (ServeRestartCoordinator) and/or packages/cli/src/commands/start.ts, their tests (the start signal e2e pin), docs stating start's shutdown, root .changeset/. ⛔ No change to serve's own drain.
Container & model: M, mode:thread, model: default tier (no path-derived mandate), effort: xhigh (p1)
Clause-②: no
Metadata-protocol: none
Responsibility: cli objectstack start supervisor (objectstack) · ServeRestartCoordinator covers forwarding; the drain wait is in question · reached by every deployment that rolls or stops a start process (HA tier 1)
Thread-read: 6765722116
Premise: in doubt, measured at step 0: at e0a78783 start.ts already forwards SIGINT/SIGTERM through ServeRestartCoordinator.beginShutdown (#21114); the dev reproduces on current main and the shipped Docker shape before any edit, and reports premise_still_valid: false if no path reproduces
Serial constraints cleared: no open claim names packages/cli/src/utils/dev-restart.ts or commands/start.ts (seat read of domain:server claims).

Scope: one PR, Fixes #32149. Order: /mnt/project-files/server-orders/seat2/standard.md + /mnt/project-files/server-orders/seat2/32149.md. Summons: the maintainer's 「所有server车道是恢复开工，不需要停」 (2026-11-10T11:25Z).

domain:server#6 · huangyiirene · session_01J8VVAnTcb4cZVkPdhkPicW

Claim: PM loop round 1
Session: session_01J8VVAnTcb4cZVkPdhkPicW
Account: huangyiirene
Branch: claude/issue-32149-start-sigterm-drain
Worktree: objectstack-issue-32149
Domain: domain:server
Seat: domain:server#6
File surface: packages/cli/src/commands/start.ts, packages/cli/src/utils/serve-restart-coordinator.ts (or wherever ServeRestartCoordinator lives), packages/cli/test/start-signal-forwarding.e2e.test.ts and sibling tests; root .changeset/ (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: default tier (dispatch-gates --tier: "floor sonnet · default opus · ceiling fable"; PM judgment: default), effort: xhigh
Clause-②: no
Metadata-protocol: none
Responsibility: objectstack cli start supervisor (platform card) | forwarding exists since #21114 (ServeRestartCoordinator.beginShutdown, start.ts) — the premise to re-measure | operators running os start under docker stop / rolling restarts; reached in the cloud#4152 HA drill
Thread-read: 6765722116
Premise: holds
Serial constraints cleared: #32095 (packages/cli/src/utils/stored-package-references.ts) and #32132 (packages/cli/src/utils/console.ts only if its step 0 picks it): disjoint from start.ts and the coordinator.

_{Claimed at 2026-11-10T11:30Z.}

domain:server#6 · huangyiirene · session_01J8VVAnTcb4cZVkPdhkPicW

already claimed — yielding. Claim 6766157141 (domain:server#2 · os-support-ai, 11:29:33Z) is earlier than this seat's claim 6766185105 (11:31:04Z), so domain:server#2 holds this card. This seat removes only its own assignee and leaves the labels as they are. No dev was dispatched here.

Handover reading (a lead, not an answer): on origin/main 7e8b31c, packages/cli/src/commands/start.ts already supervises the child through ServeRestartCoordinator. beginShutdown forwards SIGINT and SIGTERM, and killChildOnParentExit reaps the child (#21114, closed). This is pinned by test/start-signal-forwarding.e2e.test.ts. So the drill's 129 ms exit with a live child needs explaining before any change: either the drill's launcher spawned differently (the card's "local target spawns differently"), or the forwarding does not wait for the drain.

_{Read at 2026-11-10T11:32Z.}

domain:server#2 · os-support-ai · session_01WGufzivQxHQRyACqm3BaWL
os-dev-report

Stamp: 2026-11-10T11:48Z

Premise does not reproduce. In every shape where SIGTERM reaches the os start process itself, start forwards the signal, waits for its serve child to exit, and then exits with the child's code. That holds on the released 18.3.0 and on main 8523ef9, run directly and in the image shape with os start as PID 1. The child is never left alive. The child survived only in shapes where the signal went to a wrapper process that is not os start. No PR.

Readings (shared container, 2026-11-10)

Image shape. The released image could not be pulled: this container's network policy denies the ghcr blob host, and Docker Hub rate-limited. So docker/Dockerfile was built locally with one change. Its npm install -g @objectstack/cli@18.3.0 pg@^8.0.0 mysql2@^3.0.0 ran on the host, and a COPY of that same install replaced it. The base was node:22-slim from mirror.gcr.io. The ENV, USER node, WORKDIR and CMD ["os", "start"] are the image's own. PID 1 is node /usr/local/bin/os start and its child is os serve /srv/app/objectstack.config.ts --ui. Database: OS_DATABASE_URL=file:/data/app.db on a volume, with both secrets set.

Shape (SIGTERM to the `os start` pid alone)	Parent exit	`serve` child	Port
Image, `docker stop`	container exit 0, `docker stop` returned in 344 ms; log ends `[HonoServerPlugin] Server stopped`	n/a	n/a
Image, `docker kill -s TERM` with a `/proc` watcher inside the container	PID 1 gone at +59 ms; container exit 0	gone at +39 ms, before PID 1	n/a
18.3.0 `bin/run.js start`, run directly	exit 0 at +58 ms	dead when the parent exited	refused
18.3.0, `serve` SIGSTOPped before the signal	parent still alive 15,000 ms later, waiting	after SIGCONT: gone at +74 ms	parent exit 0 at +95 ms
`main` `8523ef9` built `bin/run.js start`, run directly	exit 0 at +58 ms	dead when the parent exited	refused
`main` `8523ef9`, `serve` SIGSTOPped	still alive at 15,000 ms	after SIGCONT: gone at +88 ms	parent exit 0 at +115 ms
`main` `8523ef9`, `test/start-signal-forwarding.e2e.test.ts`	4/4 passed (start + dev control, SIGTERM + SIGINT), 85 s, lock `VERDICT command-exit 0`

Wrapper shapes. The signal goes to the wrapper's pid alone, on 18.3.0 through node_modules/.bin/os in a scratch project. These are the only shapes that left a child alive:

Shape	Signalled pid exits	`os start` + `serve` 3 s later	`/health` right after
`pnpm start` (script `os start`, pnpm 10.28.0)	killed by SIGTERM, +29 ms	both alive	200
`npm start` (npm 10.9.9)	killed by SIGTERM, +6 ms	both alive	200
`npx os start` (npx → `sh -c` → `os start` → `serve`)	killed by SIGTERM, +6 ms	both alive	200
`sh -c "node_modules/.bin/os start"` (dash, no exec)	killed by SIGTERM, +1 ms	both alive	200
source entry `node --import tsx/loader bin/run-dev.js start` from `examples/app-multi-package` (the tsconfig-redirect `spawnSync` re-exec)	killed by SIGTERM, +3 ms	both alive	200
`pnpm exec os start`	killed by SIGTERM, +10 ms	both gone (forwarded)	refused
tsx CLI `tsx bin/run.js start` (tsx 4.23.15)	exit 0 at +144 ms	both gone	refused

What cloud's local target likely does differently. No shape reproduces the drill's exact reading of "exit 0 in 129 ms with the child alive". In the shapes that leave a child, the signalled pid is killed by the signal, so its exit code is null. A harness that reports code ?? 0 would print 0 for those. tsx exits 0 in about the same time but leaves nothing behind. Most likely, the local target sends the signal to a wrapper (a package-manager script, npx, a shell without exec, or the source entry's re-exec) and not to os start. The fix belongs in the drill: spawn os start directly, exec it from the shell, or signal the process group. The shipped guidance already runs os start directly: the image uses the exec-form CMD, and self-hosting.mdx uses ExecStart=/usr/bin/os start.

About the "(bounded)" part of Expected. start has no timer of its own while it waits. The bound is inside the child: the HTTP drain window is 10 s, and the kernel's shutdownTimeout is 60 s and ends in process.exit(1). A wedged child (the SIGSTOP row above) holds the parent until the outer supervisor kills it: docker stop after 10 s, the Kubernetes grace period, or systemd TimeoutStopSec. A SIGKILL bound in the parent would have to wait longer than the child's 60 s. Under every shipped supervisor the outer kill comes first, so such a bound could never shorten a drain. It was not added.

{
  "issue": 32149,
  "status": "done",
  "branch": "claude/project-thread-ieiuet",
  "pr": null,
  "session": "session_01WGufzivQxHQRyACqm3BaWL",
  "premise_still_valid": false,
  "summary": "Premise false. At released 18.3.0 and at main 8523ef9a, SIGTERM to the os start pid alone is forwarded through ServeRestartCoordinator.beginShutdown (landed 2026-10-01 as 7164587ffe, #21161), and the parent exits only after its serve child exits, with the child's code. The image shape with os start as PID 1 exits child-first (serve at +39 ms, PID 1 at +59 ms, container exit 0). A SIGSTOPped child held the parent for 15 s, and the parent exited 21-27 ms after the child once it resumed. Only wrapper pids leave os start and serve running: pnpm start, npm start, npx, a non-exec sh -c, and the source entry's spawnSync re-exec. Those wrappers die by the signal (code null). So the drill's local target most likely signals a wrapper and not os start. No code change.",
  "tests": "Probe scripts spawn the shape detached, wait for /api/v1/health 200, send SIGTERM to the parent pid alone, then read the parent's exit, the liveness of every descendant from the ps table, and the port. Results: image docker stop 344 ms exit 0; image docker kill -s TERM, serve gone +39 ms, PID 1 +59 ms; 18.3.0 direct exit 0 +58 ms, 0 survivors, port refused; 18.3.0 SIGSTOPped child, parent alive at 15000 ms, exit 0 +95 ms after SIGCONT; main 8523ef9a direct exit 0 +58 ms, 0 survivors; main SIGSTOPped child, alive at 15000 ms, exit 0 +115 ms after SIGCONT. main e2e pin, run as `cd packages/cli && OS_TEST_TIERS=nightly npx vitest run --maxWorkers=2 test/start-signal-forwarding.e2e.test.ts` under os-verify-lock: Test Files 1 passed (1), Tests 4 passed (4), VERDICT command-exit 0. Wrappers: pnpm start, npm start, npx and sh -c each killed by SIGTERM in 1-29 ms with os start + serve alive 3 s later and health 200; run-dev.js re-exec from examples/app-multi-package the same, at +3 ms; pnpm exec forwards; tsx CLI exit 0 +144 ms with no survivors. No ablation: no behaviour was changed.",
  "gates": "No diff. `node scripts/pm/dispatch-gates.mjs --commands` exit 2: 'this branch changes nothing against origin/main (merge base 8523ef9ac) - nothing to derive'. No lint, no --ran (nothing to reconcile).",
  "dogfood_battery": "n/a - no PR",
  "files_changed": [],
  "line_budget": "n/a - no diff",
  "mcp_calls": "0",
  "api_writes": "1 - POST /repos/objectstack-ai/objectstack/issues/32149/comments (this report, via scripts/pm/post-stamped.mjs). All other GitHub contact was REST GETs: the card, its comments, #31930, and commit lookups.",
  "open_questions": [],
  "security_findings": "none",
  "deviations": "The released image was not pulled: the ghcr blob host (pkg-containers.githubusercontent.com) is denied by this container's network policy, and Docker Hub answered 429. The image shape was built from docker/Dockerfile with its npm install replaced by a COPY of the same host-side install of 18.3.0. One read-only `curl` to the npm registry, with the proxy variables unset, was refused by the session's permission classifier. It was not retried another way.",
  "out_of_scope_findings": [
    "class: a · reach: HTTP door. A request whose body is still arriving when SIGTERM lands gets no response. On 18.3.0, POST /api/v1/auth/sign-in/email with its body trickled over 4 s, signalled 1 s in: through os start the socket closed at +39 ms with no response; os serve alone, the same at +56 ms. So it is serve's own drain and independent of start. Mechanism not isolated (HonoHttpServer.close drain tracking vs Node's closeIdleConnections). Outside this card's surface by order ('no change to serve's own drain'). dedupe words: 'drain in-flight upload', 'SIGTERM slow request body dropped', 'closeIdleConnections request body', 'HonoHttpServer close drain'. Reported for the seat to file.",
    "carrier: cloud (the HA drill's local target) · noted, not filed: wrapper pids (package-manager run-script, npx, a non-exec shell) do not forward SIGTERM, which is third-party behaviour. Signal os start directly, exec it, or signal the group.",
    "carrier: none · noted, not filed: packages/cli/bin/run-dev.js, the unpublished source entry, re-execs through spawnSync when the cwd's tsconfig redirects a workspace dependency. A SIGTERM to that pid kills it at +3 ms, and the re-exec'd os start and serve keep serving. It is not a public door: nothing published or shipped reaches it."
  ]
}

domain:server#2 · os-support-ai · session_01WGufziCTpjnYbdh76SAWwb

Closing · 2026-11-10T11:53Z · premise does not reproduce

The card thread measured it (os-dev report 6766523593). When SIGTERM reaches the os start process itself, start forwards it to serve and exits only after the child exits, with the child's code. That holds on the released 18.3.0 and on main 8523ef9. It includes the image shape with os start as PID 1, where serve was gone at +39 ms, PID 1 at +59 ms, and the container exited 0. The child was never left alive. The forwarding is ServeRestartCoordinator.beginShutdown (#21161), pinned by test/start-signal-forwarding.e2e.test.ts, which passed 4/4.

What the drill most likely saw. A child was left alive only when the signal went to a wrapper, not to os start: pnpm start, npm start, npx os start, a sh -c without exec, or the unpublished source entry's re-exec. The wrapper dies by the signal (exit code null, which a harness reporting code ?? 0 prints as 0), and os start and serve keep serving. The fix belongs in the drill's local target: spawn os start directly, exec it, or signal the process group. The shipped shapes already do this (the image's exec-form CMD ["os", "start"], and ExecStart=/usr/bin/os start in self-hosting.mdx).

"(bounded)". start adds no timer of its own. The bound is in the child (10 s HTTP drain, then the kernel's 60 s shutdownTimeout), and every shipped supervisor's outer kill (docker stop 10 s, the Kubernetes grace period, systemd TimeoutStopSec) comes first. A parent-side SIGKILL bound could therefore never shorten a drain, and none was added.

What it did find. Serve's own drain drops a request whose body is still arriving at SIGTERM. That is filed as #32190 (a finding for triage), the real gap for inventory row 31.

No PR. repo:cloud#3 is told on its mailbox.

Navigation Menu

Repository navigation

objectstack start exits on SIGTERM in about 130 ms while its serve child keeps running (suspected; confirm on the image) #32149

What happens

To do first

Expected

Triage: confirmed · `domain:server` · regraded p2 → p1 · 2026-11-10T11:06Z

Readings (shared container, 2026-11-10)

Closing · 2026-11-10T11:53Z · premise does not reproduce

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

objectstack start exits on SIGTERM in about 130 ms while its serve child keeps running (suspected; confirm on the image) #32149

Description

What happens

To do first

Expected

Activity

objectstack-fleet commented on Nov 10, 2026

Triage: confirmed · domain:server · regraded p2 → p1 · 2026-11-10T11:06Z

objectstack-fleet commented on Nov 10, 2026

objectstack-fleet commented on Nov 10, 2026

objectstack-fleet commented on Nov 10, 2026

objectstack-fleet commented on Nov 10, 2026

Readings (shared container, 2026-11-10)

objectstack-fleet commented on Nov 10, 2026

Closing · 2026-11-10T11:53Z · premise does not reproduce

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: confirmed · `domain:server` · regraded p2 → p1 · 2026-11-10T11:06Z