Repository navigation
Conversation
CI Test ResultsRun: #44814888958 | Commit:
No CI outcome reports were found for this run. Updated: 2026-11-20 20:53:20 UTC |
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/154349646 |
ad202a1 to
44a17e3
Compare
This comment has been minimized.
This comment has been minimized.
6d59cf5 to
73e3af6
Compare
73e3af6 to
ebab4c4
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class TaskBlock event recording to the Java profiler so known blocking intervals can be emitted directly (without relying on wall-clock sampling signals), including native/JFR plumbing and updates to wall-precheck behavior and tests.
Changes:
- Introduces public Java API
beginTaskBlock(int)/endTaskBlock(long, long, long)and native recording pipeline fordatadog.TaskBlock. - Reworks wall-precheck suppression around lifecycle-owned blocked runs, including unfiltered registry tracking and bounded “lazy backfill” candidate selection.
- Expands unit/integration/benchmark coverage for unfiltered precheck, restart safety, JVMTI delegation, and TaskBlock behavior.
Reviewed changes
Copilot reviewed 51 out of 51 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/WallclockMitigationsCombinedTest.java | Updates combined mitigation assertions to ensure context-scoped threads remain sampled and suppression counters don’t change. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java | Adds integration coverage for owned-block prechecks when filter= samples every thread. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java | Adds restart/epoch safety coverage for unfiltered tracking across multiple recordings. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/TaskBlockAssertions.java | Adds JMC-based assertion helpers for datadog.TaskBlock contract validation. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java | Updates precheck tests to new suppression semantics/counters and explicit filter= behavior. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckEfficiencyTest.java | Adjusts workload tests to align with new lifecycle-owned suppression and explicit unfiltered sampling. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java | Adds JVMTI-delegated stack path coverage for unfiltered owned-block precheck. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedPrecheckTest.java | Aligns JVMTI precheck tests with explicit filter= commands. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockPreExistingThreadTest.java | Verifies TaskBlock TLS init for threads created before profiler startup. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockDisabledTest.java | Verifies TaskBlock API stays inactive when not in all-thread scope. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockApiTest.java | End-to-end coverage for paired TaskBlock API, nesting/token rules, context admission, vthreads, and live dump behavior. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java | Ensures unsupported J9 wall engine doesn’t activate unfiltered precheck tracking. |
| ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerApiSurfaceTest.java | Verifies TaskBlock API is public while internal hooks remain non-public. |
| ddprof-test/src/test/java/com/datadoghq/profiler/context/OtelContextStorageModeTest.java | Ensures trace context is cleared during cleanup (ordering with reset). |
| ddprof-test/src/test/java/com/datadoghq/profiler/context/AllNativeContextTest.java | Ensures trace context is cleared during cleanup (ordering with reset). |
| ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java | Adds beforeProfilerStart() hook and clears trace context on stopProfiler(). |
| ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java | Exposes package-scoped hooks for benchmarking owned-block precheck overhead. |
| ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java | Adds JMH benchmark for steady-state wall timer overhead vs. owned-block population size. |
| ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp | Adds engine capability and filter= parsing regression tests. |
| ddprof-lib/src/test/cpp/wallClockCounters_ut.cpp | Renames/updates suppression counter tests to owned-block suppression. |
| ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp | Adds unit tests for bounded candidate selection/backfill logic. |
| ddprof-lib/src/test/cpp/threadFilter_ut.cpp | Expands ThreadFilter tests for registry, epoch, token encoding, and owned-block suppression eligibility. |
| ddprof-lib/src/test/cpp/taskBlockRecorder_ut.cpp | Adds unit tests for TaskBlock eligibility, rotation, and record/stack failure accounting. |
| ddprof-lib/src/test/cpp/park_state_ut.cpp | Updates tests to owned-block state semantics and slot-reset expectations. |
| ddprof-lib/src/test/cpp/jvmSupport_ut.cpp | Adds unit tests for platform vs. virtual thread classification via JNI table probing. |
| ddprof-lib/src/test/cpp/frame_ut.cpp | Adds unit test for copyJvmtiFrames() union-overlap safety. |
| ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java | Adds public TaskBlock API and native bindings; clarifies internal hook comments. |
| ddprof-lib/src/main/cpp/wallClockCounters.h | Renames suppressed-sample counter to owned-block suppression counter. |
| ddprof-lib/src/main/cpp/wallClockCandidateSelector.h | Introduces bounded randomized candidate selection with precheck-rejection backfill. |
| ddprof-lib/src/main/cpp/wallClock.h | Integrates candidate selector + new suppression counters and visit budgeting. |
| ddprof-lib/src/main/cpp/wallClock.cpp | Reworks precheck suppression to use ThreadFilter-owned block eligibility and optional lazy backfill/lookup accounting. |
| ddprof-lib/src/main/cpp/threadLocalData.h | Adds per-thread TaskBlock lifecycle state in TLS. |
| ddprof-lib/src/main/cpp/threadFilter.h | Adds registry/unfiltered tracking, epoching, owned-block snapshots/tokens, and suppression eligibility checks. |
| ddprof-lib/src/main/cpp/threadFilter.cpp | Implements registry/TID index, recording epochs, retirement, and owned-block suppression candidate logic. |
| ddprof-lib/src/main/cpp/taskBlockRecorder.h | Adds TaskBlock eligibility checks and recording helper APIs. |
| ddprof-lib/src/main/cpp/taskBlockRecorder.cpp | Implements duration thresholding and basic eligibility helpers. |
| ddprof-lib/src/main/cpp/profiler.h | Adds TaskBlock recording entry points and rotation/inflight coordination. |
| ddprof-lib/src/main/cpp/profiler.cpp | Adds TaskBlock recording implementation, rotation coordination, registry bootstrap for existing threads, and updated filter init semantics. |
| ddprof-lib/src/main/cpp/jvmThread.h | Adds capability query for native thread-id lookup support. |
| ddprof-lib/src/main/cpp/jvmThread.cpp | Implements native thread-id lookup capability helper. |
| ddprof-lib/src/main/cpp/jvmSupport.h | Declares isPlatformThread() helper for virtual thread rejection. |
| ddprof-lib/src/main/cpp/jvmSupport.cpp | Implements JNI table probing to detect virtual threads (JDK 19+). |
| ddprof-lib/src/main/cpp/jfrMetadata.h | Adds new JFR type id for TaskBlock. |
| ddprof-lib/src/main/cpp/jfrMetadata.cpp | Registers datadog.TaskBlock event metadata fields. |
| ddprof-lib/src/main/cpp/javaApi.cpp | Adds native implementations for TaskBlock API + registry-aware filter/add/remove and owned-block hooks. |
| ddprof-lib/src/main/cpp/frames.h | Adds copyJvmtiFrames() helper for overlapping JVMTI/ASGCT frame buffers. |
| ddprof-lib/src/main/cpp/flightRecorder.h | Adds TaskBlock recording entry points in recording/recorder. |
| ddprof-lib/src/main/cpp/flightRecorder.cpp | Implements TaskBlock event serialization and recorder entry. |
| ddprof-lib/src/main/cpp/event.h | Adds TaskBlock event struct; updates wall epoch suppressed counter field name/type usage. |
| ddprof-lib/src/main/cpp/engine.h | Adds wall-engine capability query for unfiltered precheck support. |
| ddprof-lib/src/main/cpp/counters.h | Adds counters for registry lookup budget/retirement and TaskBlock outcomes; renames suppression counter. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark Results (commit ebab4c4)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125876908 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10352 ms (21 iters) | ≈ +0.4% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8971 ms (24 iters) | ✅ 8823 ms (24 iters) | ≈ -1.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5997 ms (33 iters) | ✅ 5974 ms (33 iters) | ≈ -0.4% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5455 ms (36 iters) | ✅ 5469 ms (36 iters) | ≈ +0.3% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2671 ms (69 iters) | ✅ 2788 ms (67 iters) | 🔴 +4.4% | — / — |
| fj-kmeans | 25 | ✅ 2765 ms (68 iters) | ✅ 2778 ms (67 iters) | ≈ +0.5% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2054 ms (91 iters) | ✅ 2127 ms (88 iters) | 🔴 +3.6% | — / — |
| future-genetic | 25 | ✅ 2027 ms (91 iters) | ✅ 2069 ms (90 iters) | ≈ +2.1% (±2.8%) | — / — |
| naive-bayes | 21 | ✅ 1284 ms (133 iters) | ✅ 1244 ms (137 iters) | ≈ -3.1% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1013 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16291 ms (15 iters) | ✅ 15943 ms (15 iters) | ≈ -2.1% (±8.5%) | — / — |
| reactors | 25 | ✅ 18609 ms (15 iters) | ✅ 18693 ms (15 iters) | ≈ +0.5% (±4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1885 / 2016 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 2056 / 2299 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8787 / 8984 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8287 / 8556 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1235 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 1295 / 1292 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 3001 / 3006 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2834 / 2887 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3513 / 3477 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 3500 / 3460 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1612 / 1583 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1916 / 1954 | ✅ / ✅ | ✅ / ✅ |
ebab4c4 to
19cbe47
Compare
Benchmark Results (commit 19cbe47)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125884549 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10377 ms (21 iters) | ✅ 10184 ms (21 iters) | ≈ -1.9% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8790 ms (24 iters) | ✅ 8896 ms (24 iters) | ≈ +1.2% (±9.7%) | — / — |
| finagle-chirper | 21 | ✅ 5965 ms (33 iters) | ✅ 5956 ms (33 iters) | ≈ -0.2% (±25.2%) | |
| fj-kmeans | 21 | ✅ 2696 ms (70 iters) | ✅ 2843 ms (66 iters) | 🔴 +5.5% | — / — |
| fj-kmeans | 25 | ✅ 2796 ms (67 iters) | ✅ 2746 ms (68 iters) | ≈ -1.8% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2052 ms (90 iters) | ✅ 2099 ms (89 iters) | ≈ +2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2047 ms (91 iters) | ≈ +0.6% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1224 ms (139 iters) | ✅ 1250 ms (136 iters) | ≈ +2.1% (±33.2%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (168 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.6%) | — / — |
| reactors | 21 | ✅ 16927 ms (15 iters) | ✅ 15621 ms (15 iters) | 🟢 -7.7% | — / — |
| reactors | 25 | ✅ 17980 ms (15 iters) | ✅ 17858 ms (15 iters) | ≈ -0.7% (±6.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1969 / 1961 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 2168 / 2325 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 8522 / 8623 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 8617 / 8387 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1273 / 1252 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1256 / 1264 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2942 / 2959 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2855 / 2852 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3466 / 3494 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3466 / 3473 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1660 / 1623 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1789 / 1796 | ✅ / ✅ | ✅ / ✅ |
19cbe47 to
25ba2a3
Compare
Keep bounded diagnostic receipts in a process-lifetime ring so recorder transitions remain observable across recording teardown. Add stage probes and paired-API identity proofs, with JNI snapshots and raw evidence persistence for tests. Compile diagnostic transport, storage and arguments out of release builds.
Keep the debug dump driver outside the timer epoch writer's exclusive recorder window. Release epoch ownership for the real dump and restore it afterward, including exceptional exits. Preserve production try-and-drop sampling and the existing Java oracle.
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations. Applies to #663.
There was a problem hiding this comment.
More details
TaskBlock recording includes native APIs, JFR events, wall-precheck integration and DEBUG diagnostics. Stop now preserves thread references while reclaiming full-table capacity and retrying retained tails, resolving the last open finding; earlier stack-truncation, weight-retention and OpenJ9 gating fixes remain intact.
🤖 Bits Code Review · Commit e37124f
|
In response to the stacked findings, #915 carries the fixes; they are not all in #664 itself. 9d8228a76 recovers abandoned intervals; 684a12a77 fails closed for unverified non-HotSpot JNI19+ classification; 58d73c761 and 9bbd0fbf0 assert duration and retained weight. 8b6707d94 counts traced skips; 588104e50 and 73dd597e9 reject and count invalid thresholds; b626adff6 documents deferred recovery and updates test support. These changes still require the intended stack to be merged. |
What does this PR do?:
Adds the
datadog.TaskBlockJFR event and the public paired APIJavaProfiler.beginTaskBlock()/endTaskBlock(token, blocker, unblockingSpanId). One event describes one completed blocked interval. The exit hook captures the stack synchronously.WallClockBlockTrackerseparates owned blocked time from ordinary wall samples. Stale or wrong-thread tokens cannot close a newer run.TaskBlockRecorderchecks eligibility at exit: unfiltered wall precheck, no active span, no context-window crossing, and a duration of at least 1 ms.stop()flushes them before engine teardown.TaskBlockDiagnosticsreceipt ring that records each recorder decision outsideRecording. Release builds compile it out.Motivation:
A known blocking interval should not need repeated wall signals to show its elapsed time. One completed-at-exit event gives the exact duration and the blocking stack. Context-scoped profiling keeps its ordinary samples.
A missing event in a decoded JFR file does not tell why it is missing. The DEBUG receipts show where the native path stopped (admission, ownership, stack capture or serialization). They do not allocate or log on signal paths.
Additional Notes:
datadog.TaskBlock, JFR type idT_TASK_BLOCK = 133. Fields:startTime,duration,eventThread,blocker,unblockingSpanId,stackTrace,observedBlockingState,spanId,localRootSpanId, plus context attributes.datadog.WallClockSamplingEpochrenamesnumSuppressedSampledRuntonumSuppressedOwnedBlock. The counterwc_signals_suppressed_sampled_runbecomeswc_signals_suppressed_owned_block. Consumers must handle the rename.0/falseon virtual threads and outside unfiltered wall precheck.recordTaskBlock()takes the recording stripe with try-and-drop, like the sample paths. A busy stripe or a failed serialization drops the event and incrementstask_block_record_failed.endTaskBlock()then returnsfalse. Callers must not treat an event as guaranteed.stop()writes pending unowned wall tails after the signal and TaskBlock drains, with bounded retries. It recovers full JFR thread tables without a dump. A tail it still cannot write is dropped, its weight is added towc_unowned_tail_dropped_at_stop, and teardown continues. Why: one unwritable tail must not leave the profiler half stopped.wall_block_tracker, becausewallclockreads 0 when idle. The stop-time thread-ID archive is charged tojfr_buffersand freed with the recording.JfrIntegration.md) charged toNM_MISC, plus a 40-byte payload per paired-API owner. Receipts are lossy under contention;BUFFER_ACCEPTEDmeans serialization returned, not consumer delivery.lightweight=truewall sampling keep unit weight for unowned blocks: they have no reusable local trace ID for a deferred tail.human_review:mandatory). Reason: new public Java API and JFR event schema, plus lifecycle and atomic changes on signal-reachable wall-clock paths.How to test the change?:
Added and changed coverage:
taskBlockRecorder_ut(eligibility, threshold, activity admission and rotation drain, stale tokens),taskBlockDiagnostics_ut(ring publication, gaps, overwrite, exhaustion),wallTailRecording_ut(tail retention across dumps and teardown),flightRecorder_wallEpoch_ut(stop-time flush, drop counting, thread-table capacity recovery),flightRecorder_result_ut,wallClockBlockTracker_ut,wallClockTestEpochGuard_ut,threadLocalDataPool_ut,jvmSupport_ut.JavaProfilerTaskBlockApiTest(all event fields, nested and invalid tokens, live dump, bounded retries for best-effort rejection),JavaProfilerTaskBlockDisabledTest,JavaProfilerTaskBlockLightweightTest,JavaProfilerTaskBlockPreExistingThreadTest,TaskBlockRecorderTestSupportTest,UnownedBlockedTailDumpTest,LiveHeapThenUnownedBlockedTailDumpTest,UnownedWallTailContextTransitionTest.ParkBlockChurnAntagonistTest,WallClockPrecheckOverheadBenchmarkTest.Run:
./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug :ddprof-lib:gtestRelease./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='*TaskBlock*'and the same with:ddprof-test:testRelease./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='*Unowned*'For Datadog employees:
dd:platform-security-reviewskill, or file a request via the PSEC review form).bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!