Skip to content

fix(build-tree): exclude node_modules at any depth (#764) - #769

Merged
MendixMau merged 2 commits into
masterfrom
triage/764-build-tree-nested
Nov 16, 2026
Merged

MendixMau merged 2 commits into
masterfrom
triage/764-build-tree-nested

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Fixes #764

Direct lane

What changed and why (one paragraph):
project-bin/build-tree.py applied skip() only to top-level names (if top and skip(e.name)). So EXCLUDE's node_modules was dropped at the project root and nowhere else. A custom pluggable widget's source folder with its own node_modules (1.2 GB in the field) was copied into .mxtk/build/model/ on every full build, and one exec stalled for 13+ minutes.

The fix adds EXCLUDE_ANY_DEPTH = {"node_modules"}, which is skipped at every level. Below the top level, a skipped name is not in want, so the existing deletion pass in sync_dir removes a nested node_modules that an older copy already holds. No separate cleanup is needed.

The other excluded names (deployment, docs, sources, .mxcli, .mxtk, .mpr-snapshots, the VCS folders, *.mpr.lock, MXTK_BUILD_TREE_EXCLUDE) stay top-level only, as before. They are root-level toolkit or project-output state, and a widget may legitimately hold its own docs/ or deployment/. The top-level deletion guard, which keeps the copy's own deployment/, is unchanged. The docstring states the rule and its reason.

Impact

Row Answer
Process none: no human step or gate added or removed; nothing new blocks
Dependencies project-bin/build-tree.py only (installed via install-manifest.sh, picked up by sync-project.sh). No shared lib touched, no new reads, no new tools. Python 3 stdlib, as before
Context cost none: no routed or always-loaded file touched
Behaviour n/a: scripts only
Runtime Saves time on any project with a nested node_modules. Measured on a synthetic model with a 4.9 MB nested widgets/W/node_modules (400 files), on a 4-core Linux container: first sync 135 ms → 31 ms, resync 37 ms → 24 ms. At field scale (1.2 GB) this was minutes per exec. Time cost: per exec · −0.1 s at 5 MB nested node_modules (Linux container, 4 cores); minutes at the field's 1.2 GB · can force rework: n · offset: removes copying nested node_modules

Fixture result (bash tests/wave2/test-build-tree.sh <project-bin>):

  • master's build-tree.py (the rest of project-bin/ from this branch): PASS=58 FAIL=2. The two failures are B4c widgets/W/node_modules was copied and B4c a stale nested node_modules lingers in the copy.
  • this branch: PASS=60 FAIL=0.

The new case B4c checks that:

  • widgets/W/src/W.tsx is copied and widgets/W/node_modules is not;
  • a nested docs/ (a top-level-only name) is still copied;
  • a stale widgets/W/node_modules planted in the copy is removed on the next sync, while the widget sources stay.

Other checks:

  • bin/check-scripts.sh: 137/137 shell, 30/30 Node.
  • bin/check-portability.sh: clean, 390 files.
  • Leak guard (LEAKGUARD_ALLOW_NO_DENYLIST=1 LEAKGUARD_BASE=origin/master): clean.

Field evidence: a Greenfield build in auto mode (Mendix 11.15.0, toolkit ff26aa1, 2026-11-13). A nested widget node_modules of 1.2 GB was copied on every exec; the first baseline step took 870 s, and one exec stalled for 13+ minutes. The builder patched it locally. This branch has not been run on that project. It is verified by the fixture and the synthetic timing above.

  • No client data anywhere in the diff
  • Size cap: 3 files changed, about 40 lines
  • Test tier reached: T1 (the scoped test-build-tree.sh fixture, failing on master and passing here). T2 is the relayed field report, not a re-run of this branch
  • Changed instrument: the input is a directory tree, not a tool-output capture, so a golden capture doesn't apply. The change is layout-independent: sync_dir recurses from the model directory, so single-tree and two-tree behave the same, and B3 still covers two-tree. It uses only os.scandir, with no platform-specific calls
  • For a new skill: n/a
  • CHANGELOG.md line appended, crediting a Greenfield field run (2026-11-13)
  • New bug entries: n/a

Review: pending independent review


Generated by Claude Code

skip() ran only for top-level names, so a custom widget's nested
widgets/<W>/node_modules (1.2 GB in the field) was copied into
.mxtk/build/model/ on every full build; one exec stalled 13+ minutes.
node_modules is now skipped at any depth; below the top it is not in
`want`, so the existing deletion pass removes a stale copy on the next
sync. The other excluded names stay top-level only (a widget may hold
its own docs/ or deployment/).

test-build-tree.sh B4c: fresh sync and stale-copy removal; 58/2 on
master's build-tree.py, 60/0 on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
…769)

JavaScript actions' npm dependencies live in javascriptsource/<m>/**/node_modules
and mxbuild bundles them; the any-depth exclusion dropped them, so a gate build
in the copy could fail on a healthy model. That one top-level tree now keeps
its node_modules. B4c adds a copy + resync check (2 FAIL on the previous head,
62/62 now).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX: excluding node_modules at every depth also drops javascriptsource/<module>/**/node_modules. Those folders hold the npm dependencies of JavaScript actions, which mxbuild bundles, so the copy's build can now fail even when the model is fine.

Value: fixes #764 as reported. project-bin/build-tree.py:53 skips EXCLUDE_ANY_DEPTH below the top level, and the existing deletion pass (:77-84) removes a nested copy left by an older sync. Small, and not overbuilt. Keeping the other excluded names top-level only (:24-25) is right.

Regression risk: Mendix keeps JavaScript action npm dependencies inside the model tree, under javascriptsource/<m>/actions/node_modules. The toolkit already says so at pipelines/mendix-mpr/export-module-mpk.py:49-50, and Marketplace modules ship them there too.

  • Reproduced on a synthetic tree: base copies javascriptsource/mymod/actions/node_modules/big-js/index.js; head drops it.
  • On an existing project, the next sync after sync-project.sh also deletes those folders from .mxtk/build/model/. The gate build in the copy then fails on an unresolved import while an in-place build passes, so a gate verdict flips to FAIL.
  • The docstring's own warning (:13-14, "a copy that leaves out a folder mxbuild reads fails the build falsely") describes exactly this case.
  • Both layouts are affected. It is Python stdlib only, with no portability concern.

Correctness: apart from the above, the logic is right. B4c tests what it claims, and a directory fixture is acceptable here because there is no tool output to capture. But no case covers a node_modules the build actually needs, which is how the gap got through. The scoped fixture gives PASS=60 FAIL=0. No conflict with master.

Context cost: 66,196 → 66,196 (no routed file touched).

Time cost: none added per row; the copy only gets smaller.

Exact fixes required:

  1. In sync_dir, do not apply EXCLUDE_ANY_DEPTH inside the top-level javascriptsource/ subtree, and state the carve-out in the docstring and the CHANGELOG line.
  2. Add a B4c check that javascriptsource/M/actions/node_modules/x/index.js is copied and survives a resync.

Generated by Claude Code

@MendixMau

Copy link
Copy Markdown
Owner Author

Both fixes from the review are pushed in 1778580. sync_dir passes a keep_nm flag down the top-level javascriptsource/ tree, so JavaScript action node_modules are copied, while every other nested node_modules is still dropped. The docstring and the CHANGELOG line state the carve-out.

New B4c checks: javascriptsource/mymod/actions/node_modules/big-js/index.js is copied, and it survives a resync. On the previous head these 2 checks fail; now the fixture gives 62/62. check-scripts is clean.

This needs a re-review before merge.


Generated by Claude Code

@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE. Re-review: both earlier fixes landed and are correct. javascriptsource/**/node_modules is copied again, and B4c proves it on a first sync and on a resync.

Value: Fixes #764 as reported.

  • Nested node_modules is excluded at any depth (project-bin/build-tree.py:55), and an old nested copy is removed by the existing deletion pass (:77-84).
  • Fix 1 landed: keep_nm is passed down from the top-level javascriptsource entry (:50, :63), so JavaScript action npm dependencies are copied again.
  • The docstring states the carve-out and its reason (:29-31), and so does the CHANGELOG line.
  • Nothing is overbuilt.

Regression risk:

  • In both layouts, src is the model directory, so javascriptsource/ is a top entry whether the .mpr is at the root or under app/. The carve-out applies to both.
  • Top-level node_modules is still excluded through skip() when top is true (:55).
  • A project already synced by 829e5c3 gets its javascriptsource node_modules back on the next sync.
  • No exit codes or output format changed.
  • Python stdlib only, so nothing for bash 3.2 or Git Bash.
  • Only the lowercase name javascriptsource matches, which is what Mendix writes.

Correctness: Fix 2 landed. tests/wave2/test-build-tree.sh:147-156 plants javascriptsource/mymod/actions/node_modules/big-js/index.js and asserts it after a sync and after a resync.

  • Fixture run in a detached worktree: head PASS=62 FAIL=0.
  • Positive control, with 829e5c3's build-tree.py and the rest of project-bin/ from head: PASS=60 FAIL=2. The 2 failures are exactly the two new checks.
  • A directory fixture is fine here, since there is no tool output to capture.
  • No conflict with origin/master.

Context cost: 66196 → 66196 always-loaded words, measured with render-routing.sh --check on master and on head.

Time cost: per exec, the copy only gets smaller, except under javascriptsource/, which is copied as before #764. Nothing new per build row. No field re-run of this branch.


Generated by Claude Code

@MendixMau
MendixMau merged commit 1e208e7 into master Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] build-tree.py copies nested node_modules (a custom widget's 1.2 GB) into .mxtk/build on every exec; one exec stalled 13+ min

2 participants