Repository navigation
Conversation
OtelProfilesBuilder collects any number of normalized profiles, such as the time and heap profiles of one collection cycle, into one ExportProfilesServiceRequest of pprof-format/otel. All of them share the request's dictionary, as the OTel format intends. Each profile is added through a ProfileVisitor from visitor(), so whatever drives the visit, the normalizer or a profiler emitting samples itself, needs no knowledge of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are shorthands for a request with a single profile. - An OTel profile has a single sample type, so each added profile becomes one OTel profile per sample type (sample count, wall and CPU time; in-use and allocated objects and space). When every sampling event is a single timestamped observation, as with samplePerContext, the sample count is folded into the profile of the period type (wall time) instead, provided no other sample adds an observation to that profile. It then gets an observation for every sampling event, even one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time (see #199), thereby stay in the wall profile as timestamped evidence that sampling happened, while their CPU time stays in the CPU profile. No count profile then has to repeat every sample just to carry 1s. When some sampling events are summed, the count keeps its profile, so no count is lost. - The profiles' time range starts at the start of collection, so sample timestamps fall within [time_unix_nano, time_unix_nano + duration_nano), as the format asks. - Frames become locations, with columns, and the normalizer's stacks become the stack table. Functions are identified by what they encode as, display name and resolved file, so one script mapping to several original files keeps them apart. Functions, locations, stacks, attributes and links are deduplicated across all profiles. Every dictionary table starts with its zero value. - Labels become interned attributes: strings, integers, or doubles for fractional numbers, which pprof labels can't hold. An optional extractLink callback can turn some labels into a link to a span; the builder validates the ID lengths, copies the IDs, and treats a link whose trace ID or span ID is all zeros, which is invalid, as no link. - Samples with the same stack, attributes and link are merged. Timestamped samples accumulate a value and a timestamp per observation; others are summed. Other zero values, such as zero CPU time, are left out of their sample type's profile. - Frames whose source map is declared but missing locally, the candidates for server-side unminification, get a datadog.frame.missing_source_map location attribute. The profiles with such frames get a datadog.profile.has_missing_source_maps attribute, which the builder learns from the visitor's end() callback. pprof carries this as a profile comment, but OTel's pprof.profile.comment is reserved for human-readable text. No semantic convention exists for this, so the keys are provisional until the Datadog backend confirms them. Visitors run synchronously, and build() can be called after any number of completed visits; a visit that throws leaves partial data, so the builder should then be discarded. The builder isn't exported from the package yet, so that the API can settle first. pprof-format/otel first shipped in pprof-format 2.4.0, so that is now the minimum version.
Overall package sizeSelf size: 2.94 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | pprof-format | 2.4.0 | 912.36 kB | 912.36 kB | | source-map | 0.8.0 | 185.66 kB | 185.66 kB | | node-gyp-build | 4.8.4 | 13.86 kB | 13.86 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
These pass the normalized profile to a caller-supplied ProfileVisitor, instead of serializing it as pprof. The caller chooses the output, e.g. a visitor of an OtelProfilesBuilder that collects all profiles of a collection cycle into one request. They are the format-neutral counterparts of time.stop() and heap.profile(). Both use the eager stop paths of time.stop() and heap.profile(), not the lazy, callback-based ones, which are opt-in because of a race in V8 when deleting a CPU profile (#329). Both return the VisitResult.
szegedi
force-pushed
the
szegedi/otel-serializer
branch
from
November 10, 2026 11:36
c6ec787 to
59f320e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Adds an OpenTelemetry profiles serializer, built as a
ProfileVisitoron the normalizer from #430, and encoding withpprof-format/otel(pprof-format 2.4.0).Multiple profiles (both time and heap) can be collected into a single
ExpoerProfilesServiceRequestsharing the request's dictionary.Each profile is added through
builder.visitor(), so whatever drives the visit needs no knowledge of OTel.serializeTimeProfileOtel()andserializeHeapProfileOtel()are shorthands for a single-profile request.time.stopAndVisit()andheap.visitProfile()pass the normalized profile to a caller-supplied visitor instead of serializing it as pprof. They're the format-neutral counterparts oftime.stop()andheap.profile()Motivation:
Step 2 of emitting the OpenTelemetry profiles signal from the Node.js profiler: a second encoder that reuses the profile semantics from #430 instead of duplicating them.
Additional notes:
samplePerContext, the sample count is folded into the wall-time profile, which gets an observation per sampling event. Idle samples, which deliberately have zero wall time (see Preserve idle samples with zero wall time and no CPU time rollup #199), stay as timestamped evidence that sampling happened, and we don't have to have a separate count profile that has to repeat every sample just to carry 1s.extractLinkcallback can turn labels into a span link; all-zero trace or span IDs are treated as no link.datadog.frame.missing_source_maplocation attribute, and their profiles adatadog.profile.has_missing_source_mapsattribute. (pprof carries this as a profile comment, for lack of better solution.) The backend will need to be adjusted to recognize these.AI generated PR description
Adds an **OpenTelemetry profiles serializer**, built as a `ProfileVisitor` on the format-neutral normalizer from #430, and encoding with `pprof-format/otel` (pprof-format 2.4.0).OtelProfilesBuilder(ts/src/otel-serializer.ts) collects any number of normalized profiles, e.g. the time and heap profiles of one collection cycle, into oneExportProfilesServiceRequest. All of them share the request's dictionary, as the OTel format intends. Each profile is added throughbuilder.visitor(), so whatever drives the visit needs no knowledge of OTel.serializeTimeProfileOtel()andserializeHeapProfileOtel()are shorthands for a single-profile request.time.stopAndVisit()andheap.visitProfile()pass the normalized profile to a caller-supplied visitor instead of serializing it as pprof. They're the format-neutral counterparts oftime.stop()andheap.profile(), and use the same eager stop paths. The lazy, callback-based paths stay opt-in because of the V8CpuProfiledeletion race (Revert lazy profile-tree changes to mitigate CpuProfile::Delete crash #329).Mapping highlights:
samplePerContext, the sample count is folded into the wall-time profile, which gets an observation per sampling event. Idle samples, which deliberately have zero wall time (see Preserve idle samples with zero wall time and no CPU time rollup #199), thereby stay as timestamped evidence that sampling happened, and no count profile has to repeat every sample just to carry 1s.[time_unix_nano, time_unix_nano + duration_nano), since the time range starts at the start of collection.extractLinkcallback can turn labels into a span link; all-zero trace or span IDs are treated as no link.Motivation:
Step 2 of emitting the OpenTelemetry profiles signal from the Node.js profiler: a second encoder that reuses the profile semantics from #430 instead of duplicating them.
Additional Notes:
datadog.frame.missing_source_maplocation attribute, and their profiles adatadog.profile.has_missing_source_mapsattribute. pprof carries this as a profile comment, but OTel'spprof.profile.commentis reserved for human-readable text. No semantic convention exists, so these keys are provisional until the Datadog backend confirms them.How to test the change?:
Unit tests in
ts/test/test-otel-serializer.tscover the mapping, deduplication across profiles, sample merging, links and the missing-source-map attributes, and decode the encoded request to check it round-trips.ts/test/test-time-profiler.tsandts/test/test-heap-profiler.tscoverstopAndVisit()andvisitProfile(), including state after a throwing visitor.npm test.