Skip to content

Count a heap snapshot as 1.5x the heap in the room check - #1818

Merged
helpcodeai merged 2 commits into
mainfrom
fix/heap-snapshot-headroom
Nov 21, 2026
Merged

helpcodeai merged 2 commits into
mainfrom
fix/heap-snapshot-headroom

Conversation

@helpcodeai

Copy link
Copy Markdown
Contributor

Summary

The heap snapshot guard (HEAP_SNAPSHOT_PERCENT) allowed a snapshot whenever rss + heapUsed fit in 85 % of the cgroup limit, on the assumption that a snapshot needs about one extra heap of memory. That was too little. On 2026-11-21, with a 2.4 GB heap and 2.7 GB RSS in a 6 GB container, the guard allowed the snapshot (5.1 GB fits under 5.2 GB). The process then grew to 5.9 GB, 1.32x the heap, and the kernel killed it. The setting is off by default and is off again where it was turned on. This PR makes the guard refuse a case like this one.

Changes

  • process-vitals.ts: new SNAPSHOT_COST_FACTOR = 1.5 and snapshotCost(). The room check now counts 1.5x the used heap.
  • process-vitals.service.ts: the "snapshot" and "snapshot-skipped" log lines quote that estimate.
  • process-vitals.spec.ts: a regression test with the 2026-11-21 numbers, which now give snapshot-skipped.
  • .env.example: the comment on HEAP_SNAPSHOT_PERCENT explains the 1.5x.

Self-hosted and Cloud behave the same. The default is unchanged (0, no snapshot).

Type

  • Bug fix

Testing

  • Added/updated unit tests

In packages/backend:

  • npx jest --maxWorkers=2 src/common/process-vitals: 29/29 pass
  • npx eslint on the three files: clean
  • npx tsc --noEmit -p tsconfig.json: no errors in the changed files. This worktree had no generated Prisma client, so unrelated files report Prisma errors; CI generates it.

Checklist

  • My code follows the existing code style
  • I have tested my changes locally
  • I have updated documentation (if applicable)
  • This PR has a descriptive title

The guard let a snapshot through when rss + heapUsed fit in 85 % of the
cgroup limit. On 2026-11-21 a snapshot of a 2.4 GB heap with 2.7 GB RSS
grew the process to 5.9 GB (1.32x the heap) and the kernel killed it in
its 6 GB cgroup. The check now counts 1.5x the used heap, and the log
lines quote that estimate.
@helpcodeai
helpcodeai enabled auto-merge (squash) November 21, 2026 14:51
@helpcodeai
helpcodeai merged commit 4e38d32 into main Nov 21, 2026
16 checks passed
@helpcodeai
helpcodeai deleted the fix/heap-snapshot-headroom branch November 21, 2026 15:18
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant