Skip to content

fs: add diagnostics_channel tracing - #66575

Open
timfish wants to merge 6 commits into
nodejs:mainfrom
timfish:fs-diagnostics-channel
Open

timfish wants to merge 6 commits into
nodejs:mainfrom
timfish:fs-diagnostics-channel

Conversation

@timfish

@timfish timfish commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

This adds a diagnostics_channel TracingChannel for each node:fs operation. It is a different approach to #65370, based on the review feedback there.

Each operation publishes to tracing:fs.<operation>:*, for example tracing:fs.stat:start. The sync, callback and promise forms of an operation share a channel, so a subscriber to fs.readFile sees fs.readFile(), fs.readFileSync(), fsPromises.readFile() and filehandle.readFile(). The context has api ('sync', 'callback' or 'promise'), args (a copy of the arguments as passed), and the leading arguments by name, such as path, fd and dest. The docs have the full list.

The events are published from JS with traceSync(), traceCallback() and tracePromise(), so all events of one call share one context object, error is always followed by end or asyncEnd, asyncStart has result, and bindStore() works. Buffers and the other arguments are passed by reference in args. The readFile and writeFile one-round-trip paths, fsPromises.readFile(), readFileSync() with an encoding, existsSync() and the FileHandle methods all publish.

Each traced function starts with one line, for example if (shouldTraceFs('stat')) return traceFsCallback('stat', stat, this, arguments);. When the channel has subscribers, the helper calls the same function again inside the trace, and a flag makes that inner call skip the check. The whole body then runs inside the start scope. This matters for bindStore(): the FSReqCallback or binding promise must be created inside the scope, because a request created before it does not carry the store to the callback. Nested fs calls also run inside the parent's store. Because the check is in the function body, captured references and ESM named imports still publish. When nothing subscribes, the rest of the body is the same as on main.

Calls with invalid arguments publish start, error and end. Calls that a VFS handler serves also publish. fsPromises.glob(), fs.watch(), fs.watchFile() and dir.read() do not publish. An operation that uses other fs operations publishes those too, inside its own events. For example, fs.readFileSync() without an encoding publishes fs.open, fs.read and fs.close inside fs.readFile.

With plain compare.js, the async and stream benchmarks in benchmark/fs do not change, but some short sync benchmarks are slower. With 15 runs and no subscribers, fstatSync was about 5% slower, statSync about 3%, and failing readFileSync calls 5% to 9%. Most of this is the cost of TracingChannel.hasSubscribers before V8 optimizes it. Many of these benchmarks do 10,000 calls in a new process, so most checks run while the getter is still in the interpreter or in Sparkplug. There, the getter calls BoundedChannel.hasSubscribers and Channel.hasSubscribers for all five channels, which costs 60 to 100 ns per check. Once it is optimised, it costs about 3 ns. Next to a 300 to 900 ns sync call, the difference shows.

Once hasSubscribers has been optimized, calling it becomes insignificant apart from sync calls that throw. These are 1-3% slower. When an exception passes through a function that runs as Sparkplug code, V8 finds the bytecode offset with a linear walk from the start of the function (Code::GetBytecodeOffsetForBaselinePC, from Isolate::Throw and LookupExceptionHandlerInTable). So any code before the binding call makes each throw slower. A single if (x.active !== 0) return; costs about 50 ns per failing openSync() there. I did not find a way around this.

I did not change the benchmarks to warm the hasSubscribers getters. A follow-up could make TracingChannel.hasSubscribers cheaper before optimisation.

The new tests cover the event order, the shared context, bindStore() across callbacks, promises and nested calls, and the FileHandle methods. A coverage test fails if a public fs function does not publish, or publishes more than once.

Fixes: #65330
Refs: #65370
CC: @nodejs/diagnostics


This PR was partly AI generated but everything was reviewed by a water based human

@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to file-system APIs and the fs module. needs-ci PRs that need a full CI run. labels Oct 7, 2026
@timfish
timfish force-pushed the fs-diagnostics-channel branch from f73bebb to f7a0dc8 Compare October 7, 2026 18:21
@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.22485% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.52%. Comparing base (df2ea49) to head (87bf499).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
lib/fs.js 98.10% 8 Missing ⚠️
lib/internal/fs/promises.js 97.10% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #66575      +/-   ##
==========================================
+ Coverage   90.51%   90.52%   +0.01%     
==========================================
  Files         797      797              
  Lines      278252   279085     +833     
  Branches    53472    53694     +222     
==========================================
+ Hits       251868   252650     +782     
- Misses      16761    16820      +59     
+ Partials     9623     9615       -8     
Files with missing lines Coverage Δ
lib/internal/fs/dir.js 94.36% <100.00%> (+0.52%) ⬆️
lib/internal/fs/utils.js 96.67% <100.00%> (+0.37%) ⬆️
lib/internal/fs/promises.js 91.74% <97.10%> (+0.51%) ⬆️
lib/fs.js 97.49% <98.10%> (+0.13%) ⬆️

... and 29 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hamidrezaghavami hamidrezaghavami left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation in lib/fs.js and lib/internal/fs/promises.js is incredibly clean. I didn't review the C++ bindings, but LGTM on the JS/Promises side!

Comment thread lib/fs.js Outdated
logaretm added a commit to getsentry/js-tracing-channels-proposals that referenced this pull request Oct 13, 2026
Comment thread doc/api/diagnostics_channel.md Outdated
Comment thread doc/api/diagnostics_channel.md Outdated
Comment thread doc/api/diagnostics_channel.md Outdated
Comment on lines +1720 to +1721
`fsPromises.glob()`, `fs.watch()`, `fs.watchFile()`, and `dir.read()` do not
publish events.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not?

@Qard Qard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than having the getFsChannel, shouldTraceFs, and the traceFs* functions, we should be defining the channels at the top level. By using these functions which take a string name for the channels at publish time you're eliminating the performance benefit diagnostic channels have of splitting the lookup cost out of the hot publishing path, leaving it with basically the same performance you'd get with a plain event emitter.

The diagnostics channel design is intended to be used with explicit top-level defined channels which V8 can then optimize the calling of as simply a held pointer from the outer scope which is very fast to reference as V8 will optimize it, rather than doing channel lookups at every publish or check.

timfish added a commit to timfish/node that referenced this pull request Oct 20, 2026
The checks looked each channel up by name on every call, through a
dictionary and a helper that took the operation name. The channels and
their context field names are now module-level constants, one tracer
object per operation, so a check is a property read on a held object.

Refs: nodejs#66575 (review)
Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
@timfish
timfish requested review from Qard and jsumners-nr October 20, 2026 11:07

@Qard Qard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this FsTracer wrapper type rather than just using the tracing channel instance directly? You're wasting a bunch of computation on every publish of recomputing the message shape for each channel when that would not change between publishes to that channel. Seems to me like it'd be better to just use the tracing channel instances directly, have each publish site craft its own particular shape based on the fields relevant to that channel, and use hasSubscribers in place of the shouldTrace wrapper function. I don't really see a reason for wrapping like this. 🤔

timfish added a commit to timfish/node that referenced this pull request Oct 20, 2026
The fs modules now hold the TracingChannel instances themselves, and
each publish site builds its own context literal, so the shape of the
context is fixed per site instead of computed from a field table on
every publish. The check at the start of each function is
`hasSubscribers` on the held channel.

The trace helpers call the traced function again with a private `this`,
so that the second entry skips the check. This replaces the re-entry
flag and the wrapper that consumed it. No traced function uses `this`
for anything else.

Refs: nodejs#66575 (review)
Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
@timfish

timfish commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Done in c9571ff.

Each operation now has a TracingChannel constant in internal/fs/utils (fsStat = tracingChannel('fs.stat') and so on), each call site builds its own context literal, and the check is this !== kTraced && fsStat.hasSubscribers. The helpers call the function again with kTraced as this, which replaces the module flag and its try/finally.

Cost of the check at a call site, with the previous two versions for comparison. The main column is the same loop calling a function that only returns false, since main has no check:

Tier main (no check) name lookup module-level tracer this change
Interpreter 18 ns 146 ns 124 ns 109 ns
Sparkplug 10 ns 80 ns 78 ns 67 ns
Maglev 0.6 ns 7.0 ns 5.1 ns 4.7 ns
TurboFan ~1 ns 2.4 ns 1.2 ns ~1 ns

Against main, the fs benchmarks for the callback and promise forms and for successful sync reads and writes are within noise. Sync calls that throw are 2 to 10% slower, which I think is the cost of the extra frame when Sparkplug walks the bytecode to build the stack trace, not the check itself. Startup is unchanged.

@timfish
timfish requested a review from Qard October 20, 2026 12:43
Each node:fs operation publishes to its own TracingChannel, named
fs.<operation>. The sync, callback, and promise forms of an operation
share a channel, and the event context carries the form in `api`, the
caller's arguments in `args`, and the leading arguments by name.

The events are published from JavaScript with the TracingChannel
helpers, so all events of a call share one context object, stores bound
with bindStore() reach the callback and nested operations, and the
error event is always followed by end or asyncEnd.

Each traced function starts with a check that calls the function again
inside the trace when the channel has subscribers. The whole body then
runs inside the start scope, so requests capture the bound stores, and
captured function references still publish.

Assisted-by: claude:opus-5.5
Signed-off-by: Tim Fish <tim@timfish.uk>
The tracing checks of realpath.native and realpathSync.native passed
the functions to the trace helpers as property reads. When user code
replaces the property, the trace helper called the replacement again,
so it ran twice for each call. The functions now pass themselves by
their own name.

The new checks assert from setImmediate(), because an exception thrown
in a realpath.native() callback or in a nextTick from it is lost. That
is a separate bug.

Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
Say which events each form of an fs operation publishes, and explain
the function names in the channel table.

Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
The checks looked each channel up by name on every call, through a
dictionary and a helper that took the operation name. The channels and
their context field names are now module-level constants, one tracer
object per operation, so a check is a property read on a held object.

Refs: nodejs#66575 (review)
Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
The fs modules now hold the TracingChannel instances themselves, and
each publish site builds its own context literal, so the shape of the
context is fixed per site instead of computed from a field table on
every publish. The check at the start of each function is
`hasSubscribers` on the held channel.

The trace helpers call the traced function again with a private `this`,
so that the second entry skips the check. This replaces the re-entry
flag and the wrapper that consumed it. No traced function uses `this`
for anything else.

Refs: nodejs#66575 (review)
Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
fs.mkstemp(), fs.mkstempSync() and fsPromises.mkstemp() landed after
this branch, so the coverage test failed on them. Add the fs.mkstemp
channel with the same context as fs.mkdtemp.

Assisted-by: claude:fable-5.1
Signed-off-by: Tim Fish <tim@timfish.uk>
@timfish
timfish force-pushed the fs-diagnostics-channel branch from c9571ff to 87bf499 Compare October 20, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fs Issues and PRs related to file-system APIs and the fs module. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

diagnostics_channel: add a channel for filesystem operations

6 participants