Skip to content

[Installer] Replace weighted memory split with standard per-RAM profiles (16/20/24/32/48/64 GB) #2951

Open
Open
[Installer] Replace weighted memory split with standard per-RAM profiles (16/20/24/32/48/64 GB)#2951

Description

@karlbrown87
karlbrown87
opened on Oct 21, 2026

Summary

Follow-up to #2918. That issue (and its fix in #2929) focused on how the installer's weighted BalanceMemory logic behaves on large hosts, mainly the 64 GB case. The problem is not limited to one server size, though: every host size needs a sane, predictable allocation.

Instead of tuning priorities and caps until the weighted split comes out right, we propose that the installer use fixed standard memory profiles for 16, 20, 24, 32, 48 and 64 GB hosts. These profiles come from a sizing study we ran, and production v11 servers are already running with exactly these values and working correctly. The values below should be used as-is.

Problem

  • The weighted split (installer/system/memory.go + MinMemory/MaxMemory in installer/docker/stack.go) produces numbers that are hard to predict and hard to reason about per host size. It took a code change (fix(installer): rebalance memory — cap EP manager, scale backend + JVM heap (#2918) #2929) just to fix the 64 GB case.
  • The OpenSearch JVM heap is always set to 50 % of the node1 container (compose.go: -Xms = -Xmx = opensearchMem/2), regardless of host size.
  • web-pdf can be sized below what Chromium needs to render reports. Below ~1,250 MB, report generation has failed.

Proposed memory profiles

Container memory limits per service, in MB (1 GB = 1024 MB):

Service 16 GB 20 GB 24 GB 32 GB 48 GB 64 GB
frontend 100 100 100 100 100 100
backend 1,536 2,048 2,048 2,048 4,096 6,144
event-processor-manager 2,560 3,072 3,072 4,096 7,168 8,192
event-processor-worker 3,072 4,596 4,596 8,192 12,480 12,480
node1 – OpenSearch (container) 6,144 7,168 10,240 14,336 20,480 32,768
OpenSearch JVM heap (Xms = Xmx) 4,096 5,540 8,192 10,240 16,384 26,624
agentmanager 350 350 350 1,024 1,024 1,024
user-auditor 350 350 350 512 512 512
web-pdf 1,250 1,250 1,250 1,250 1,250 1,250
postgres 512 512 512 512 512 512
Total limits 15,874 19,446 22,518 32,070 47,622 62,982
Host RAM 16,384 20,480 24,576 32,768 49,152 65,536
Free for the OS 510 ⚠️ 1,034 2,058 698 ⚠️ 1,530 2,554

The OpenSearch heap is a JVM setting inside the node1 limit and is not added to the total. ⚠️ = less than 1 GB left for the OS.

OpenSearch (node1) gets the largest share: 35–50 % of host RAM.

Note: cloud-heavy deployments — invert manager and worker.
event-processor-manager ingests the cloud integrations (Azure, GCP, Office 365, AWS and the other vendor input plugins). event-processor-worker processes everything else (Linux and Windows agents, syslog, etc.).
The table assumes the usual case, where most of the log volume comes from agents, so the worker gets the larger share. When a customer has more cloud services than agent-based sources, swap the two values: give the worker's memory to the manager and the manager's memory to the worker. The total stays the same.

Changes needed in the installer

  1. Profile-based allocation. Add the table above as standard profiles (e.g. in installer/docker/stack.go or a new installer/system/profiles.go). Pick the profile from detected RAM: the largest profile that is ≤ the host RAM, with a small tolerance, because the RAM the OS reports is usually a bit below the nominal size (a "64 GB" server often reports ~62–63 GB). Keep the current BalanceMemory logic only as a fallback for hosts outside the 16–64 GB range.
  2. OpenSearch heap per profile. Replace the fixed opensearchMem/2 in installer/docker/compose.go with the heap value from the profile (OPENSEARCH_JAVA_OPTS=-Xms<heap>m -Xmx<heap>m). Validate that heap < node1 limit before writing the compose file.
  3. web-pdf floor. Never assign less than 1,250 MB to web-pdf.
  4. Low-headroom warning. The 16 GB and 32 GB profiles leave less than 1 GB for the OS. The installer should log a warning when the selected profile leaves < 1 GB free.
  5. --invert flag for cloud-heavy deployments. Swap the memory between event-processor-manager and event-processor-worker without changing the total (see the note above). Example for the 64 GB profile: manager 12,480 MB / worker 8,192 MB.
  6. Apply to existing installations. utmstack_installer --recalculate-memory should apply the profile to running installs: back up the compose file before writing, redeploy with the installed version, and print a rollback command if the deploy fails.

Notes

Acceptance criteria

  • A fresh install on a 16 / 20 / 24 / 32 / 48 / 64 GB host gets exactly the limits in the table.
  • The OpenSearch heap matches the profile, and heap < node1 limit is enforced.
  • web-pdf is never below 1,250 MB.
  • A warning is printed when < 1 GB is left for the OS.
  • --recalculate-memory applies the profile to existing installations, with a compose backup and a rollback path.

Related: #2918, #2929, #1856

Activity

  1. osmontero commented on Oct 22, 2026

    @osmontero
    osmontero
    on Oct 22, 2026
    Member

    Thanks for the sizing study. I went through this against the current source (installer/docker/stack.go, installer/docker/compose.go, installer/system/memory.go) and re-ran the allocation for the six sizes. Here's where I land.

    Short version: the profile table is a design preference, not a bug — but there is one real defect in the current code it exposes, and two defects in the table itself. I'll fix the code defect as a small standalone change; the table needs the two issues below addressed before I'd adopt it.

    What is a real defect in the current code (I'll fix this)

    web-pdf can be sized below what Chromium needs. stack.go has web-pdf MinMemory: 1024, and the README minimum host is 16 GB. On a 16 GB install the weighted split can assign web-pdf less than ~1,250 MB, and headless Chrome will OOM printPage below that. This is independently corroborated by #2934, where web-pdf OOM'd at ~1.1 GiB. Fix: floor web-pdf at 1,250 MB (MinMemory 1024 → 1250). I'll land that as its own small PR.

    What is NOT a defect in the current code

    • The weighted split produces valid, non-crashing allocations at every size in your table (16/20/24/32/48/64 GB): never below a service's MinMemory, never above MaxMemory, and it always fits the host. It differs from your numbers, but it's not broken. "Hard to predict" is a fair maintainability point, but it's a preference, not a bug.
    • OpenSearch heap = 50% of the container (compose.go:372, -Xms = -Xmx = opensearchMem/2) is standard OpenSearch sizing (≥50% of RAM to the JVM). It's not size-appropriate, just not wrong.

    Two problems in the proposed table

    1. The 16 GB and 32 GB profiles leave the OS less headroom than the current code. Your ⚠ flags this, but it's worth being explicit: the current allocator reserves a full 1 GB for the OS (memory.go SYSTEM_RESERVED_MEMORY = 1000). Your 16 GB profile leaves ~510 MB and your 32 GB profile leaves ~698 MB for the OS — that's below the reserve the code already enforces. Adopting the table as-is would regress OS headroom on the two sizes you've already marked ⚠.

    2. The 20 GB profile's OpenSearch is dangerously thin. node1 = 7,168 MB with a 5,540 MB heap leaves only ~1,628 MB for Lucene's off-heap (segment cache, mmap'd index, thread stacks). OpenSearch officially needs ~25–50% of RAM as off-heap headroom for exactly this; ~23% is below that. Your own Notes section says to watch this, but 20 GB is a profile you're asking me to ship, so I'd want it raised (e.g. node1 → ~8,192 MB) or the profile dropped until a 20 GB box is validated in production.

    On the feature requests

    • --invert (swap manager/worker for cloud-heavy installs): reasonable, but it's a new flag and a new code path. I'd treat it as its own enhancement, not part of a "memory fix."
    • --recalculate-memory applying profiles with backup + rollback: the flag exists and re-balances; the compose-backup + auto-rollback is new and worth doing, but again as its own change.

    Proposed path

    1. Now: web-pdf 1,250 MB floor (real defect, tiny diff).
    2. If we adopt profiles: fix (1) OS headroom on 16/32 GB and (2) the 20 GB OpenSearch off-heap, keep the current BalanceMemory as the fallback for out-of-range hosts, and validate each profile on a real instance before shipping — per your own note that the 20 GB one is tight.
    3. --invert and the recalc rollback as separate enhancements.

    Happy to take on #1 immediately.

  2. osmontero
    added a commit that references this issue on Oct 22, 2026

    Merge pull request #2956 from utmstack/fix/webpdf-memory-floor

    b1de408

  3. osmontero commented on Oct 22, 2026

    @osmontero
    osmontero
    on Oct 22, 2026
    Member

    Update: the real defect is fixed — PR #2956 (merged, in v11.3.0): web-pdf MinMemory 1024 → 1250 in installer/docker/stack.go. Simulated: on a 16 GB host web-pdf now gets 1342 MB (was 1128).

    So, in line with your message "focus on real defect only": the profile redesign (table, per-size OpenSearch heap, --invert, recalc rollback) is left open as a design discussion — I've flagged the two table issues above (OS headroom on 16/32 GB, thin off-heap on the 20 GB profile) that should be resolved before adopting it.

  4. osmontero
    added a commit that references this issue on Oct 22, 2026

    Merge pull request #2958 from utmstack/fix/installer-memory-preserve

    cd6ad13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions