Skip to content

feat(taskblock): add JVM producers and retain failed recording output (PROF-14931) - #665

Draft
kaahos wants to merge 69 commits into
taskblock-review-fixesfrom
paul.fournillon/taskblock-jvm-producers
Draft

kaahos wants to merge 69 commits into
taskblock-review-fixesfrom
paul.fournillon/taskblock-jvm-producers

Conversation

@kaahos

@kaahos kaahos commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?:

Adds automatic JVM producers for datadog.TaskBlock on top of the recorder from #664:

  • Java park hooks own park intervals. Java generic hooks own SLEEPING and OBJECT_WAIT.
  • Native JVMTI callbacks own monitor contention (CONTENDED). They also own Object.wait (WAITING), unless the Java bridge delegates monitor wait to Java instrumentation through the new JavaProfiler.getInstance(libLocation, scratchDir, delegateMonitorWaitEvents) overload.
  • Each platform thread has separate park, generic and monitor BlockSlot payloads. A nested or mismatched exit cannot consume another producer's owner.
  • Before a producer takes ownership, it writes any pending unowned wall tail.

Also changes JFR output handling (PROF-16798): a failed write, copy or close no longer loses the retained recording.

  • Recording::flush() writes disjoint ranges with bounded pwrite() retries and resets the buffer only after every byte is written.
  • dump() truncates the source only after a complete copy and a successful destination close. A failed copy keeps the sealed chunks, and a later dump delivers them.
  • A write, patch or close failure quarantines the output (sticky error, counted in jfr_output_failures).

Motivation:

JVM producers must record one completed interval per blocking call, with the entry context and state, across dumps, context changes and restarts. Dropping a pending unowned tail at ownership would lose wall time that was already observed.

Before this change, a partial copy or a failed close was treated as delivery. The profiler could then truncate or free the only copy of the recording.

Additional Notes:

  • Stack: builds on fix(taskblock): reject invalid thresholds and recover abandoned intervals (PROF-14931) #915.
  • Reviewer acknowledgment, stop with quarantined output: stop() still tears down all engines and producers. Then it keeps the recording and latches the non-restartable ERROR state; a new start() is refused. It skips releaseRecordingMemory(), so buffers, dictionary roots and uncommitted wall tails stay owned. It still closes the intern gate. On this path pending wall tails are retained, not dropped, and wc_unowned_tail_dropped_at_stop does not change. A failed close is not retried, because descriptor ownership after a failed close is platform-dependent.
  • Reviewer acknowledgment, output disk use: a failed copy keeps the sealed chunks in the source file until a later dump succeeds. The source file grows in that case.
  • Reviewer acknowledgment, ownership and timing: monitor-wait delegation is process-wide and fixed by the first bridge initialization. A later getInstance call with a different preference throws IllegalStateException. A native HotSpot wait ends at MonitorWaited, before monitor reacquisition. A delegated Java wait ends at the instrumented return from Object.wait. A live dump emits no prefix for an open interval; stop/start invalidates open owners.
  • Reviewer acknowledgment, cost: each owner carries three BlockSlot payloads. Each eligible exit captures a stack synchronously. Monitor blocker hashing runs only after the duration check passes.
  • Virtual-thread classification: park and generic hooks take virtuality from Java Thread.isVirtual() (through a MethodHandle, which avoids reflection's varargs allocation). JVMTI monitor callbacks use JNI IsVirtualThread. When that function is missing on HotSpot, the thread is rejected, task_block_virtual_thread_detection_failed is incremented and a one-time warning is logged. On non-HotSpot JDK 19+ the monitor producers record nothing.
  • JfrIntegration.md documents RecordingOutput::Delivery. It certifies native write, copy and close completion only, not consumer decoding or upload. jfr_deliveries_certified counts it.
  • Bundled supporting fixes in the diff:
    • ASan: a gtest covers freeChunks() removing poison before unmap. The production fix is main's (PROF-16142).
    • macOS OS::copyFile() uses a fixed buffer instead of mapping the source, and reports partial copies (PROF-16798).
  • Bundled test infrastructure:
    • build-and-summarize now writes deterministic summaries to build/reports/claude/gradle-summary.{md,json} (utils/gradle_summary.py) and no longer calls an agent (PROF-16796).
    • utils/run-containers-tests.sh adds --source-root, --offline, --max-workers and --artifacts-dir for bounded local runs (PROF-16797).
    • New qualify-real-vm GitHub job (debug only): HotSpot 25 and OpenJ9 17 run the producer lifecycle and classifier tests and must pass without skips (.github/scripts/assert_vm_qualification.py). Part of PROF-14931: it guards this PR's producers against tests that skip on a real VM.
    • googleJavaFormat 1.30.0 to 1.36.1, with real-formatter TestKit tests in SpotlessConventionPluginTest (PROF-16797).
  • Contracts: ProfilingEngines, JfrIntegration, JvmSupportLayer, CallTraceStorage, TestingGuide.
  • Human review: mandatory (label human_review:mandatory). Reason: changes stop()/start() lifecycle on output failure and adds JVMTI callback producers that run on arbitrary JVM threads.

How to test the change?:

Added coverage:

  • gtests: recordingOutput_ut (RecordingOutputTest: sticky errors, disjoint reservations, partial write/copy/close), flightRecorder_result_ut and flightRecorder_wallEpoch_ut (dump and start output failure branches), profilerStartRollback_ut (ProfilerStartRollbackOutputTest), wallTailRecording_ut (QuarantinedWallTailStopTest), taskBlockProducers_ut, vmEntry_ut (NativeMonitorEventsTest, ProfilerBridgeDelegationTest, MonitorEventConfigurationTest), jvmSupport_ut (JvmSupportThreadClassificationTest), park_state_ut, referenceChainJfrRoundtrip_ut, linearAllocator_nativemem_ut.
  • Java: ParkTaskBlockTest, MonitorTaskBlockTest, JvmtiBasedParkTaskBlockTest, JvmtiBasedMonitorTaskBlockTest, TaskBlockReceivingScopeTest, TaskBlockProducerCompatibilityTest, UnownedTailBeforeTaskBlockTest, BlockingHookAllocationTest, RecordingDeliveryJfrTest, JavaProfilerTest (monitor-delegation singleton conflicts and reuse).
  • Real-VM: HotSpotProducerLifecycleTest, HotSpotDelegatedProducerLifecycleTest, OpenJ9ProducerLifecycleTest, OpenJ9CpuProducerLifecycleTest, RealVmProducerTest, RealVmThreadClassifierTest.
  • Tooling: utils/test_gradle_summary.py, utils/test_run_containers_tests.py, utils/test_profiler_start_init.py, .github/scripts/tests/test_assert_vm_qualification.py, ProfilerAccessTest.

Run:

  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug :ddprof-lib:gtestRelease
  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='*TaskBlock*' and -Ptests=RecordingDeliveryJfrTest
  • Real-VM tests need the matching JDK as JAVA_TEST_HOME; the qualify-real-vm job runs them on HotSpot 25 and OpenJ9 17.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review (run the dd:platform-security-review skill, or file a request via the PSEC review form). bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-14931, PROF-16798, PROF-16796, PROF-16797

Unsure? Have a question? Request a review!

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #44815694480 | Commit: 68d7d36 | Duration: 30m 4s (longest job)

❌ 3 of 108 test jobs failed

Status Overview

JDK glibc-aarch64/asan glibc-aarch64/debug glibc-aarch64/release glibc-aarch64/tsan glibc-amd64/asan glibc-amd64/debug glibc-amd64/release glibc-amd64/tsan musl-aarch64/debug musl-aarch64/release musl-amd64/debug musl-amd64/release
8 - - - - ✅ ✅ ✅ ✅ - - - -
8-ibm - - - - ✅ ✅ ✅ ❌ - - - -
8-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
8-librca - - - - - - - - ✅ ✅ ✅ ✅
8-orcl - - - - ✅ ✅ ✅ ✅ - - - -
11 - - - - ✅ ✅ ✅ ✅ - - - -
11-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
11-librca - - - - - - - - ❌ ✅ ✅ ✅
17 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-librca - - - - - - - - ✅ ✅ ✅ ✅
21 ✅ ✅ ✅ ✅ ✅ ✅ ❌ ✅ - - - -
21-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
21-librca - - - - - - - - ✅ ✅ ✅ ✅
25 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
25-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
25-librca - - - - - - - - ✅ ✅ ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

⚠️ Flaky tests — failed, then passed on retry

Test Cells Message
UnownedWallTailContextTransitionTest.knownSpanThenClearKeepsOriginalTail__ glibc-21-release-amd64 org.opentest4j.AssertionFailedError: Coverage incomplete: no retained A tail after span clear ==> expected: <true> but w
GCGenerationsTest.shouldNotReportPreviousRecordingAdmissions__ musl-11-librca-debug-aarch64 memory=128,generations=true java.io.IOException: Error occurred while parsing JFR recording

These fail the build. Passing on a second run makes a test flaky, not passing. Fix it, or quarantine it against a ticket so the debt is tracked rather than forgotten.

Consider quarantining these — click for ready-made entries

A quarantined test still runs and still reports; its failures just stop
turning CI red. To quarantine one:

  1. Open a PROF ticket for the test, linking the failing job.
  2. Append the line below to ddprof-test/quarantine.txt, replacing
    PROF-XXXXX with the ticket number.
  3. Check the cells and reason columns — the proposal only knows what
    failed in this run, and a narrower cells glob keeps the same test
    gating everywhere it has not misbehaved.

CI fails once review_by passes, so an entry expires instead of piling up.

# test | ticket | added | review_by | cells | reason
com.datadoghq.profiler.wallclock.UnownedWallTailContextTransitionTest.knownSpanThenClearKeepsOriginalTail | PROF-XXXXX | 2026-11-20 | 2027-02-18 | glibc-21-release-amd64 | org.opentest4j.AssertionFailedError: Coverage incomplete: no retained A tail after span clear ==> expected: <true> but was: <false> (seen in: glibc-21-release-amd64)
com.datadoghq.profiler.memleak.GCGenerationsTest.shouldNotReportPreviousRecordingAdmissions | PROF-XXXXX | 2026-11-20 | 2027-02-18 | musl-11-librca-debug-aarch64 | memory=128,generations=true java.io.IOException: Error occurred while parsing JFR recording (seen in: musl-11-librca-debug-aarch64)

Retried 2 of 102 cells.

Failed Jobs

Summary: Total: 108 | Passed: 105 | Failed: 3


Updated: 2026-11-20 21:29:30 UTC

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch 2 times, most recently from 1beeda2 to 38efbf8 Compare July 16, 2026 15:20
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 38efbf8)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125051342 Commit: 38efbf8f7d94004f2ec282bc6c50148a99f67616

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +4.1% (2719→2830 ms)
  • 🔴 fj-kmeans (JDK 25): runtime +2.8% (2737→2815 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10365 ms (21 iters) ✅ 10195 ms (21 iters) ≈ -1.6% (±11.4%) — / —
akka-uct 25 ✅ 8859 ms (24 iters) ✅ 8925 ms (24 iters) ≈ +0.7% (±10.1%) — / —
finagle-chirper 21 ✅ 5945 ms (33 iters) ✅ 5953 ms (33 iters) ≈ +0.1% (±24.9%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5495 ms (36 iters) ✅ 5431 ms (36 iters) ≈ -1.2% (±24.2%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2719 ms (68 iters) ✅ 2830 ms (66 iters) 🔴 +4.1% — / —
fj-kmeans 25 ✅ 2737 ms (68 iters) ✅ 2815 ms (66 iters) 🔴 +2.8% — / —
future-genetic 21 ✅ 2114 ms (88 iters) ✅ 2140 ms (87 iters) ≈ +1.2% (±2.7%) — / —
future-genetic 25 ✅ 2080 ms (90 iters) ✅ 2126 ms (88 iters) ≈ +2.2% (±2.5%) — / —
naive-bayes 21 ✅ 1299 ms (131 iters) ✅ 1232 ms (139 iters) ≈ -5.2% (±31.9%) — / —
naive-bayes 25 ✅ 997 ms (172 iters) ✅ 1004 ms (170 iters) ≈ +0.7% (±31.8%) — / —
reactors 21 ✅ 16202 ms (15 iters) ✅ 15908 ms (15 iters) ≈ -1.8% (±7.6%) — / —
reactors 25 ✅ 18524 ms (15 iters) ✅ 18484 ms (15 iters) ≈ -0.2% (±3.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 2045 / 1958 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2353 / 2059 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 6 / 2 8437 / 8382 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / 2 8800 / 8439 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1248 / 1279 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / 1 1262 / 1271 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 2963 / 2980 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 3 2955 / 2936 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 7 3466 / 3539 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 5 / 1 3478 / 3531 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1700 / 1609 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1905 / 1873 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/154349800

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from 38efbf8 to cdb8274 Compare July 16, 2026 22:13
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit cdb8274)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125162043 Commit: cdb8274124fc1947560c5c91bbf213a248e6b608

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5.1% (2683→2821 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
finagle-chirper 21 ✅ 5956 ms (33 iters) ✅ 6031 ms (33 iters) ≈ +1.3% (±24.8%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5457 ms (36 iters) ✅ 5444 ms (36 iters) ≈ -0.2% (±23.9%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2683 ms (70 iters) ✅ 2821 ms (66 iters) 🔴 +5.1% — / —
fj-kmeans 25 ✅ 2786 ms (67 iters) ✅ 2839 ms (66 iters) ≈ +1.9% (±2.7%) — / —
future-genetic 21 ✅ 2063 ms (90 iters) ✅ 2095 ms (89 iters) ≈ +1.6% (±2.6%) — / —
future-genetic 25 ✅ 2072 ms (90 iters) ✅ 2125 ms (87 iters) ≈ +2.6% (±2.6%) — / —
naive-bayes 21 ✅ 1257 ms (135 iters) ✅ 1250 ms (136 iters) ≈ -0.6% (±32.8%) — / —
reactors 21 ✅ 16137 ms (15 iters) ✅ 15974 ms (15 iters) ≈ -1% (±7.5%) — / —
reactors 25 ✅ 18395 ms (15 iters) ✅ 18505 ms (15 iters) ≈ +0.6% (±4.4%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2260 / 2182 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8741 / 8673 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8238 / 8373 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 4 / 2 1241 / 1256 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1287 / 1282 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / 1 2971 / 2981 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 1 2968 / 2956 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 5 / 3 3468 / 3474 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 9 / 3 3464 / 3465 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 2 1531 / 1536 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1857 / 1856 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit e13ec27)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125211559 Commit: e13ec275311412f74c94ccb558f2c544418a0634

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10495 ms (21 iters) ✅ 10316 ms (21 iters) ≈ -1.7% (±10.8%) — / —
akka-uct 25 ✅ 8849 ms (24 iters) ✅ 8938 ms (24 iters) ≈ +1% (±10.2%) — / —
finagle-chirper 21 ✅ 5936 ms (33 iters) ✅ 5938 ms (33 iters) ≈ +0% (±24.9%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5534 ms (36 iters) ✅ 5474 ms (36 iters) ≈ -1.1% (±24.5%) ⚠️ W:3 / ⚠️ W:4
fj-kmeans 21 ✅ 2773 ms (67 iters) ✅ 2775 ms (67 iters) ≈ +0.1% (±2.7%) — / —
fj-kmeans 25 ✅ 2803 ms (66 iters) ✅ 2840 ms (66 iters) ≈ +1.3% (±2.6%) — / —
future-genetic 21 ✅ 2090 ms (89 iters) ✅ 2093 ms (89 iters) ≈ +0.1% (±2.7%) — / —
future-genetic 25 ✅ 2086 ms (89 iters) ✅ 2118 ms (87 iters) ≈ +1.5% (±2.4%) — / —
naive-bayes 21 ✅ 1251 ms (136 iters) ✅ 1268 ms (135 iters) ≈ +1.4% (±33.1%) — / —
naive-bayes 25 ✅ 1016 ms (169 iters) ✅ 1028 ms (166 iters) ≈ +1.2% (±31.8%) — / —
reactors 25 ✅ 18334 ms (15 iters) ✅ 18614 ms (15 iters) ≈ +1.5% (±4.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 1 1924 / 1980 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 2 2127 / 2360 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8378 / 8654 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8670 / 8433 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1259 / 1251 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1288 / 1264 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 4 2942 / 2917 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 1 2898 / 2871 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 1 3492 / 3510 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 3 3496 / 3479 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1858 / 1656 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1890 / 1893 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from e13ec27 to c111812 Compare July 17, 2026 09:43
@dd-octo-sts

dd-octo-sts Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit c111812)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125236325 Commit: c11181224884586f33d98ba19aa41e6162ecf5f2

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10288 ms (21 iters) ✅ 10150 ms (21 iters) ≈ -1.3% (±11.5%) — / —
akka-uct 25 ✅ 8875 ms (24 iters) ✅ 8975 ms (24 iters) ≈ +1.1% (±10.6%) — / —
finagle-chirper 25 ✅ 5508 ms (36 iters) ✅ 5491 ms (36 iters) ≈ -0.3% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2661 ms (69 iters) ✅ 2713 ms (69 iters) ≈ +2% (±2.7%) — / —
fj-kmeans 25 ✅ 2865 ms (66 iters) ✅ 2841 ms (66 iters) ≈ -0.8% (±2.6%) — / —
future-genetic 21 ✅ 2107 ms (88 iters) ✅ 2126 ms (87 iters) ≈ +0.9% (±2.7%) — / —
future-genetic 25 ✅ 2068 ms (89 iters) ✅ 2110 ms (88 iters) ≈ +2% (±2.6%) — / —
naive-bayes 21 ✅ 1292 ms (132 iters) ✅ 1224 ms (139 iters) ≈ -5.3% (±31.9%) — / —
naive-bayes 25 ✅ 1012 ms (169 iters) ✅ 984 ms (173 iters) ≈ -2.8% (±31.3%) — / —
reactors 21 ✅ 16274 ms (15 iters) ✅ 17426 ms (15 iters) ≈ +7.1% (±7.6%) — / —
reactors 25 ✅ 18845 ms (15 iters) ✅ 18270 ms (15 iters) ≈ -3.1% (±3.8%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / 4 2097 / 1909 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 1 2062 / 2338 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 1 / 2 8704 / 8842 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 3 / 2 8473 / 8584 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 1 1246 / 1251 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 2 1309 / 1292 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / 2 2935 / 2923 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 1 2825 / 2908 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 5 / 4 3535 / 3528 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 6 3515 / 3473 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 1 1706 / 1935 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1880 / 1862 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from c111812 to df472c0 Compare July 20, 2026 02:03
@dd-octo-sts

dd-octo-sts Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit df472c0)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125568035 Commit: df472c043ef31dc15b077cc33c5116e3b9f11e23

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5% (2692→2827 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 25 ✅ 8806 ms (24 iters) ✅ 8924 ms (24 iters) ≈ +1.3% (±11.1%) — / —
finagle-chirper 21 ✅ 6033 ms (33 iters) ✅ 5987 ms (33 iters) ≈ -0.8% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5500 ms (36 iters) ✅ 5495 ms (36 iters) ≈ -0.1% (±24.2%) ⚠️ W:3 / ⚠️ W:4
fj-kmeans 21 ✅ 2692 ms (70 iters) ✅ 2827 ms (66 iters) 🔴 +5% — / —
future-genetic 21 ✅ 2075 ms (89 iters) ✅ 2042 ms (91 iters) ≈ -1.6% (±2.6%) — / —
future-genetic 25 ✅ 2036 ms (91 iters) ✅ 2023 ms (91 iters) ≈ -0.6% (±2.8%) — / —
naive-bayes 21 ✅ 1228 ms (138 iters) ✅ 1283 ms (133 iters) ≈ +4.5% (±33.5%) — / —
reactors 21 ✅ 16202 ms (15 iters) ✅ 16354 ms (15 iters) ≈ +0.9% (±6.6%) — / —
reactors 25 ✅ 18212 ms (15 iters) ✅ 18060 ms (15 iters) ≈ -0.8% (±4.8%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 5 2176 / 2305 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 1 / 2 8602 / 8811 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8746 / 8149 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1255 / 1277 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 3 1280 / 1287 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 4 / ✅ 2966 / 2985 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 3 / ✅ 2906 / 2872 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 5 / 3 3486 / 3512 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 2 3483 / 3521 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1654 / 1756 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1822 / 1798 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from df472c0 to 46fdadd Compare July 20, 2026 08:03
@dd-octo-sts

dd-octo-sts Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 46fdadd)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125595112 Commit: 46fdadd7a212b7224c08b7dea5a0e2f8dff3166d

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10206 ms (21 iters) ✅ 10428 ms (21 iters) ≈ +2.2% (±11.8%) — / —
akka-uct 25 ✅ 8818 ms (24 iters) ✅ 8800 ms (24 iters) ≈ -0.2% (±9.8%) — / —
finagle-chirper 21 ✅ 6051 ms (33 iters) ✅ 5960 ms (33 iters) ≈ -1.5% (±25%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5475 ms (36 iters) ✅ 5507 ms (36 iters) ≈ +0.6% (±24.1%) ⚠️ W:3 / ⚠️ W:4
fj-kmeans 21 ✅ 2700 ms (70 iters) ✅ 2757 ms (68 iters) ≈ +2.1% (±2.8%) — / —
fj-kmeans 25 ✅ 2839 ms (66 iters) ✅ 2853 ms (66 iters) ≈ +0.5% (±2.5%) — / —
future-genetic 21 ✅ 2113 ms (88 iters) ✅ 2118 ms (88 iters) ≈ +0.2% (±2.7%) — / —
future-genetic 25 ✅ 2026 ms (91 iters) ✅ 2070 ms (90 iters) ≈ +2.2% (±2.7%) — / —
naive-bayes 21 ✅ 1254 ms (137 iters) ✅ 1245 ms (137 iters) ≈ -0.7% (±32.1%) — / —
naive-bayes 25 ✅ 1028 ms (166 iters) ✅ 1020 ms (168 iters) ≈ -0.8% (±31.4%) — / —
reactors 21 ✅ 16534 ms (15 iters) ✅ 16250 ms (15 iters) ≈ -1.7% (±8.7%) — / —
reactors 25 ✅ 18124 ms (15 iters) ✅ 18095 ms (15 iters) ≈ -0.2% (±4.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 5 2079 / 1934 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 2051 / 2211 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 5 8791 / 8807 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8270 / 8308 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 1 1284 / 1300 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 1 1297 / 1291 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / 1 3010 / 3036 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 2 2860 / 2882 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 4 3492 / 3544 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 6 3485 / 3508 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1656 / 1697 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 2 1826 / 1779 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from 46fdadd to 658df77 Compare July 20, 2026 14:35
@dd-octo-sts

dd-octo-sts Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 658df77)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125689745 Commit: 658df7762e09e57c0b9c835a39c42495922d564f

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +3.5% (2679→2773 ms)
  • 🔴 future-genetic (JDK 21): runtime +4% (2051→2133 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10260 ms (21 iters) ✅ 10296 ms (21 iters) ≈ +0.4% (±10.9%) — / —
akka-uct 25 ✅ 8840 ms (24 iters) ✅ 8942 ms (24 iters) ≈ +1.2% (±10.6%) — / —
finagle-chirper 21 ✅ 6010 ms (33 iters) ✅ 5968 ms (33 iters) ≈ -0.7% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5480 ms (36 iters) ✅ 5467 ms (36 iters) ≈ -0.2% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2679 ms (69 iters) ✅ 2773 ms (67 iters) 🔴 +3.5% — / —
fj-kmeans 25 ✅ 2769 ms (68 iters) ✅ 2839 ms (66 iters) ≈ +2.5% (±2.8%) — / —
future-genetic 21 ✅ 2051 ms (90 iters) ✅ 2133 ms (87 iters) 🔴 +4% — / —
future-genetic 25 ✅ 2047 ms (91 iters) ✅ 2037 ms (91 iters) ≈ -0.5% (±2.5%) — / —
naive-bayes 21 ✅ 1254 ms (136 iters) ✅ 1314 ms (131 iters) ≈ +4.8% (±33.9%) — / —
naive-bayes 25 ✅ 1015 ms (169 iters) ✅ 983 ms (173 iters) ≈ -3.2% (±31%) — / —
reactors 21 ✅ 16594 ms (15 iters) ✅ 16990 ms (15 iters) ≈ +2.4% (±9.1%) — / —
reactors 25 ✅ 18557 ms (15 iters) ✅ 18660 ms (15 iters) ≈ +0.6% (±4.3%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 1 1856 / 2016 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 1 2373 / 2253 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 5 / 7 8704 / 8448 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 2 8503 / 8277 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 2 1260 / 1244 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / 1 1282 / 1284 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 2984 / 2927 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 3 2889 / 2841 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 3 3476 / 3519 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 6 3442 / 3468 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1680 / 1865 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1868 / 1860 ✅ / ✅ ✅ / ✅

Copilot AI review requested due to automatic review settings July 21, 2026 07:37
@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from 658df77 to 746edc2 Compare July 21, 2026 07:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds first-class TaskBlock emission for common JVM blocking states (park, Object.wait, monitor contention), and reworks wall-clock precheck/suppression to use lifecycle-owned blocked intervals (instead of the prior once-per-run suppression model) while preserving context-window sampling semantics.

Changes:

  • Introduces a new datadog.TaskBlock JFR event and a public paired Java API (beginTaskBlock / endTaskBlock) plus internal park/monitor producers.
  • Implements native lifecycle ownership + unfiltered-thread registry tracking and a bounded candidate backfill strategy to prevent O(N) wall-tick regressions.
  • Adds broad Java + C++ test coverage (including restart/pre-existing-thread scenarios, JVMTI-delegated variants, and benchmark hooks).

Reviewed changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/WallclockMitigationsCombinedTest.java Updates combined mitigation test expectations to ensure context-scoped threads remain sampled.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java New integration tests for owned-block suppression behavior when filter= samples all threads.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java New tests ensuring unfiltered tracking/registry state is correctly reset across recording restarts.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/TaskBlockAssertions.java New shared assertions for the datadog.TaskBlock event contract.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java Refactors precheck tests to the new lifecycle-owned suppression model and unowned blocked weighting counters.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckEfficiencyTest.java Updates efficiency tests to match lifecycle ownership semantics and explicit unfiltered sampling configuration.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/ParkTaskBlockTest.java New tests verifying TaskBlock emission from Java-owned park hooks and suppression counters.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/MonitorTaskBlockTest.java New tests verifying TaskBlock emission from JVMTI monitor callbacks and restart recovery.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java New subclass to exercise unfiltered precheck via delegated JVMTI stack collection.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedPrecheckTest.java Updates command strings to use explicit filter= under new behavior.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedParkTaskBlockTest.java New JVMTI-delegated variant of park TaskBlock tests.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedMonitorTaskBlockTest.java New JVMTI-delegated variant of monitor TaskBlock tests.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockPreExistingThreadTest.java New test covering TaskBlock TLS initialization for threads created before profiler startup.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockDisabledTest.java New test verifying TaskBlock API stays inactive outside any sampling scope.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockApiTest.java New end-to-end tests for the paired TaskBlock API semantics and rejection rules.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java New test ensuring J9 wall sampling doesn’t activate unfiltered precheck tracking.
ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerTest.java Adds child-JVM agentpath initialization tests and pre-existing monitor callback coverage.
ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerApiSurfaceTest.java Updates API-surface assertions: TaskBlock API public, internal hooks stay non-public, monitor delegation APIs are public.
ddprof-test/src/test/java/com/datadoghq/profiler/ExternalLauncher.java Extends launcher scenarios to cover agent compatibility, delegation conflicts, and pre-existing monitor callbacks.
ddprof-test/src/test/java/com/datadoghq/profiler/context/OtelContextStorageModeTest.java Ensures trace context is cleared during cleanup alongside thread context reset.
ddprof-test/src/test/java/com/datadoghq/profiler/context/AllNativeContextTest.java Ensures trace context is cleared during cleanup alongside thread context reset.
ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java Adds beforeProfilerStart() hook and improves teardown to clear trace context reliably.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java New helper exposing package-scoped owned-block hooks to JMH benchmarks.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java New benchmark measuring wall-clock timer overhead under growing owned-block populations.
ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp Adds capability + filter parsing tests relevant to unfiltered tracking (filter omitted vs filter=).
ddprof-lib/src/test/cpp/wallClockCounters_ut.cpp Updates counter tests to the renamed owned-block suppression metric.
ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp New unit tests for randomized bounded candidate selection/backfill.
ddprof-lib/src/test/cpp/taskBlockRecorder_ut.cpp New unit tests for TaskBlock eligibility, rotation gating, and error accounting.
ddprof-lib/src/test/cpp/park_state_ut.cpp Expands state-machine tests for park and monitor snapshot/exit behavior and owned-block transitions.
ddprof-lib/src/test/cpp/jvmSupport_ut.cpp New unit tests for platform-thread classification (virtual thread rejection) across JNI versions.
ddprof-lib/src/test/cpp/frame_ut.cpp Adds unit test for safe JVMTI->ASGCT frame copying with overlapping buffers.
ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java Adds monitor-delegation-aware getInstance, TaskBlock public API, and thread-parameterized native hooks.
ddprof-lib/src/main/cpp/wallClockCounters.h Renames and redefines wall-clock suppression counter to owned-block semantics.
ddprof-lib/src/main/cpp/wallClockCandidateSelector.h New header implementing bounded randomized candidate visitation/backfill.
ddprof-lib/src/main/cpp/wallClock.h Integrates candidate selector into wall timer loop and adds engine capability declaration.
ddprof-lib/src/main/cpp/wallClock.cpp Reworks precheck logic to rely on owned-block registry suppression and bounded slot lookup backfill.
ddprof-lib/src/main/cpp/vmEntry.h Adds monitor delegation ownership model and initialization result enum.
ddprof-lib/src/main/cpp/vmEntry.cpp Implements JVMTI monitor callbacks producing TaskBlocks, plus one-time bridge ownership negotiation.
ddprof-lib/src/main/cpp/threadLocalData.h Adds per-thread state for park/monitor/taskblock lifecycles and snapshotting.
ddprof-lib/src/main/cpp/threadFilter.h Adds unfiltered registry tracking, TID index, recording epochs, and owned-block suppression candidate checks.
ddprof-lib/src/main/cpp/threadFilter.cpp Implements registry activation, per-recording epoch publication, lock-protected indexing, and slot lookup APIs.
ddprof-lib/src/main/cpp/taskBlockRecorder.h New TaskBlock recorder API + eligibility helpers + rotation-safe activity RAII.
ddprof-lib/src/main/cpp/taskBlockRecorder.cpp Implements TaskBlock duration threshold and recording/cleanup at exit paths.
ddprof-lib/src/main/cpp/profiler.h Adds TaskBlock recording API, rotation gating fields, and bootstrap registration hook.
ddprof-lib/src/main/cpp/profiler.cpp Implements TaskBlock recording through JVMTI stack capture, rotation gating, registry bootstrap for existing threads, and start/stop integration.
ddprof-lib/src/main/cpp/jvmThread.h Adds supportsNativeThreadIdLookup() used for registry bootstrap logic.
ddprof-lib/src/main/cpp/jvmThread.cpp Implements native-thread-id lookup capability check.
ddprof-lib/src/main/cpp/jvmSupport.h Adds platform-thread classification API.
ddprof-lib/src/main/cpp/jvmSupport.cpp Implements platform-thread classification (virtual-thread rejection) using JNI version + IsVirtualThread.
ddprof-lib/src/main/cpp/jfrMetadata.h Adds new JFR type id for datadog.TaskBlock.
ddprof-lib/src/main/cpp/jfrMetadata.cpp Defines datadog.TaskBlock metadata schema.
ddprof-lib/src/main/cpp/javaApi.cpp Wires new JNI entrypoints for TaskBlock API and updates park/block hooks to be thread-aware and emit TaskBlocks.
ddprof-lib/src/main/cpp/frames.h Adds copyJvmtiFrames() helper to safely convert overlapping JVMTI frame buffers.
ddprof-lib/src/main/cpp/flightRecorder.h Adds TaskBlock recording API to the JFR writer interface.
ddprof-lib/src/main/cpp/flightRecorder.cpp Implements TaskBlock event serialization and a rotation-safe record path.
ddprof-lib/src/main/cpp/event.h Adds TaskBlockEvent struct and updates wall-clock epoch suppressed counter naming.
ddprof-lib/src/main/cpp/engine.h Adds wall-engine capability hook supportsUnfilteredWallPrecheck().
ddprof-lib/src/main/cpp/counters.h Adds new counters for registry/backfill and TaskBlock lifecycle/accounting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddprof-lib/src/main/cpp/frames.h Resolved
@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 746edc2)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125879034 Commit: 746edc2a235e9d064e56b38261b8e7b630f9c846

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +4.4% (2634→2749 ms)
  • 🟢 reactors (JDK 21): runtime -10.8% (17619→15712 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10315 ms (21 iters) ✅ 10281 ms (21 iters) ≈ -0.3% (±11.1%) — / —
akka-uct 25 ✅ 8869 ms (24 iters) ✅ 8962 ms (24 iters) ≈ +1% (±10%) — / —
finagle-chirper 21 ✅ 6014 ms (33 iters) ✅ 5992 ms (33 iters) ≈ -0.4% (±25.2%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5449 ms (36 iters) ✅ 5424 ms (36 iters) ≈ -0.5% (±24%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2634 ms (72 iters) ✅ 2749 ms (68 iters) 🔴 +4.4% — / —
fj-kmeans 25 ✅ 2757 ms (68 iters) ✅ 2825 ms (66 iters) ≈ +2.5% (±2.8%) — / —
future-genetic 21 ✅ 2097 ms (88 iters) ✅ 2137 ms (87 iters) ≈ +1.9% (±2.8%) — / —
future-genetic 25 ✅ 2067 ms (89 iters) ✅ 2031 ms (91 iters) ≈ -1.7% (±2.7%) — / —
naive-bayes 21 ✅ 1253 ms (136 iters) ✅ 1253 ms (137 iters) ≈ 0% (±32.9%) — / —
naive-bayes 25 ✅ 1008 ms (169 iters) ✅ 1002 ms (170 iters) ≈ -0.6% (±31.9%) — / —
reactors 21 ✅ 17619 ms (15 iters) ✅ 15712 ms (16 iters) 🟢 -10.8% — / —
reactors 25 ✅ 18550 ms (15 iters) ✅ 18609 ms (15 iters) ≈ +0.3% (±4%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / ✅ 1987 / 1971 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 2 2310 / 2435 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 1 / 3 8921 / 8512 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 1 8447 / 8030 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 4 / 1 1274 / 1266 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1280 / 1270 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 4 / 1 2912 / 3053 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 1 2767 / 2834 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 1 3517 / 3509 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 6 / 6 3483 / 3497 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1802 / 1666 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1883 / 1920 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch 2 times, most recently from 1db6b94 to e9b2308 Compare July 21, 2026 14:33
@datadog-datadog-us1-prod

This comment has been minimized.

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from e9b2308 to cdbed88 Compare July 22, 2026 16:00
@dd-octo-sts

dd-octo-sts Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit cdbed88)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126290898 Commit: cdbed880011f03ca3bc466ddc38ba882241f4794

⚠️ Significant outliers

  • 🔴 future-genetic (JDK 25): runtime +4.1% (2023→2105 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10260 ms (21 iters) ✅ 10222 ms (21 iters) ≈ -0.4% (±11.2%) — / —
akka-uct 25 ✅ 8935 ms (24 iters) ✅ 8930 ms (24 iters) ≈ -0.1% (±10.7%) — / —
finagle-chirper 21 ✅ 6038 ms (33 iters) ✅ 5962 ms (33 iters) ≈ -1.3% (±24.4%) ⚠️ W:4 / ⚠️ W:3
finagle-chirper 25 ✅ 5468 ms (36 iters) ✅ 5427 ms (36 iters) ≈ -0.7% (±24%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2764 ms (68 iters) ✅ 2816 ms (66 iters) ≈ +1.9% (±2.7%) — / —
fj-kmeans 25 ✅ 2810 ms (66 iters) ✅ 2802 ms (66 iters) ≈ -0.3% (±2.6%) — / —
future-genetic 21 ✅ 2130 ms (87 iters) ✅ 2084 ms (89 iters) ≈ -2.2% (±2.6%) — / —
future-genetic 25 ✅ 2023 ms (91 iters) ✅ 2105 ms (89 iters) 🔴 +4.1% — / —
naive-bayes 21 ✅ 1254 ms (137 iters) ✅ 1294 ms (132 iters) ≈ +3.2% (±32.7%) — / —
naive-bayes 25 ✅ 965 ms (176 iters) ✅ 1011 ms (169 iters) ≈ +4.8% (±32.6%) — / —
reactors 21 ✅ 17025 ms (15 iters) ✅ 16431 ms (15 iters) ≈ -3.5% (±7.3%) — / —
reactors 25 ✅ 18745 ms (15 iters) ✅ 18914 ms (15 iters) ≈ +0.9% (±4.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 1951 / 2053 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2293 / 2284 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 4 8904 / 8718 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 2 8249 / 8246 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1302 / 1261 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1251 / 1245 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 5 2947 / 2965 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 6 2754 / 3026 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 6 / 8 3535 / 3486 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 3 3472 / 3491 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1677 / 1597 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 2 1961 / 1906 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-jvm-producers branch from cdbed88 to 3208f71 Compare July 22, 2026 21:31
@dd-octo-sts

dd-octo-sts Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 3208f71)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126387908 Commit: 3208f71ed42c70305b77ea2f76d8222de30bfee5

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10357 ms (21 iters) ✅ 10320 ms (21 iters) ≈ -0.4% (±11.6%) — / —
akka-uct 25 ✅ 8770 ms (24 iters) ✅ 8916 ms (24 iters) ≈ +1.7% (±10%) — / —
finagle-chirper 21 ✅ 5978 ms (33 iters) ✅ 5931 ms (33 iters) ≈ -0.8% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5468 ms (36 iters) ✅ 5475 ms (36 iters) ≈ +0.1% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2764 ms (68 iters) ✅ 2718 ms (68 iters) ≈ -1.7% (±2.7%) — / —
fj-kmeans 25 ✅ 2815 ms (66 iters) ✅ 2822 ms (66 iters) ≈ +0.2% (±2.6%) — / —
future-genetic 21 ✅ 2040 ms (91 iters) ✅ 2046 ms (90 iters) ≈ +0.3% (±2.5%) — / —
future-genetic 25 ✅ 2084 ms (89 iters) ✅ 2075 ms (89 iters) ≈ -0.4% (±2.6%) — / —
naive-bayes 21 ✅ 1264 ms (136 iters) ✅ 1278 ms (133 iters) ≈ +1.1% (±32.8%) — / —
naive-bayes 25 ✅ 1021 ms (167 iters) ✅ 958 ms (178 iters) ≈ -6.2% (±30.7%) — / —
reactors 21 ✅ 16060 ms (16 iters) ✅ 16378 ms (15 iters) ≈ +2% (±6.8%) — / —
reactors 25 ✅ 18225 ms (15 iters) ✅ 18739 ms (15 iters) ≈ +2.8% (±6.3%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 4 1979 / 1937 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 1 2352 / 2212 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8286 / 8349 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8188 / 8233 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1264 / 1289 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 5 1263 / 1285 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 2997 / 2913 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 2898 / 2891 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 2 3512 / 3481 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ ✅ / 6 3486 / 3523 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1715 / 1776 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1857 / 1912 ✅ / ✅ ✅ / ✅

@kaahos

kaahos commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@kaahos

kaahos commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you.

This run
  • 13:29 · started · CI on head aec1f679d in progress (86 pass, 85 pending; DDCI Status red, not required)
  • 13:30 · answered · 3 chatgpt-codex-connector threads (profiler.cpp start order, JavaProfiler.java getInstance delegation, threadFilter.h unowned tail) — already fixed at head
  • 13:29 · CI red · CI Run 44741962189 cancelled by the new run 44744799357 that adding the dog-sitter label triggered (same head aec1f679d); waiting on the new run
  • 13:31 · Bits · @datadog review posted for draft head aec1f679d
  • 13:42 · Bits · FAIL on aec1f679d — datadog-datadog-us1-prod thread on JavaProfilerTest.java:392 (pre-existing-thread monitor tests need the platform-thread guard on non-HotSpot JDK 19+); fixing
  • 13:50 · pushed · 77afe25 — skip pre-existing monitor tests when platform-thread classification is unsupported (code-review medium: 1 Low, not fixed)
  • 13:50 · answered · datadog-datadog-us1-prod thread on JavaProfilerTest.java:392 — fixed in 77afe25
  • 13:50 · Bits · @datadog review posted for draft head 77afe254a
  • 14:01 · CI red · test-matrix / test-linux-glibc-amd64 (17-graal, tsan, regular) — gtestTsan_elfparser_ut SIGABRT with no test output (elfparser not touched by this PR); rerun queued for when run 44748071544 finishes
  • 14:12 · Bits · FAIL on 77afe254a — 3 datadog-datadog-us1-prod threads (flightRecorder.cpp:1255 delivery receipt, MonitorTaskBlockTest.java:423 engine gate, DelegatedObjectWaitScenario.java:54 worker scope)
  • 14:19 · pushed · e1a6712 — gate monitor tests on TaskBlockAssertions.isSupported(), scope delegated-wait assertions to the worker
  • 14:19 · answered · DelegatedObjectWaitScenario.java:54 fixed; MonitorTaskBlockTest.java:423 partly fixed (Park guard waits on the author's decision); flightRecorder.cpp:1255 waits on the author's decision (options A/B posted)
  • 14:19 · Bits · @datadog review posted for draft head e1a6712d7
  • 14:19 · stopped · fix-push cap (2 rounds) reached

Stop summary. Commits pushed: 77afe25 (skip the pre-existing monitor tests where platform-thread classification is unsupported) and e1a6712 (gate the monitor tests on engine support, scope the delegated-wait assertions to the worker). Both were fast-forward pushes with no rebase. Threads left for the author: the flightRecorder.cpp:1255 delivery receipt when the next chunk fails (design choice, options posted), and the ParkTaskBlockTest guard (choose isSupported(), or an engine-only probe). CI on e1a6712 had just started. The tsan elfparser_ut SIGABRT on 77afe25 was never retried, because that head is superseded. Bits triggered on: aec1f67, 77afe25, e1a6712.

✅ stopped 14:19 UTC · fix-push cap reached; CI running on e1a6712, 2 threads await the author's decision

@kaahos

kaahos commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The pre-existing-thread monitor tests require TaskBlock events on OpenJ9 configurations where native monitor producers intentionally reject every callback, causing test failures on those runtimes.

Open Bits AI session

🤖 Bits Code Review · Commit aec1f67

Comment on lines +390 to +392
private void assertPreExistingMonitorCallback(String target, String expectedState)
throws Exception {
Path recording = Files.createTempFile("preexisting-monitor-callback-", ".jfr");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Gate native-monitor success tests on runtime support

On OpenJ9 reporting JNI 19 or later, JVMSupport::isPlatformThread() deliberately returns false, so native monitor callbacks produce no TaskBlock events. Both new pre-existing-thread tests nevertheless require a matching event, causing deterministic test failures on these runtimes. Apply the runtime capability guard used by MonitorTaskBlockTest to their shared helper.

Suggested change
private void assertPreExistingMonitorCallback(String target, String expectedState)
throws Exception {
Path recording = Files.createTempFile("preexisting-monitor-callback-", ".jfr");
private void assertPreExistingMonitorCallback(String target, String expectedState)
throws Exception {
String vmName = System.getProperty("java.vm.name", "");
assumeTrue(
!Platform.isJavaVersionAtLeast(19)
|| vmName.contains("OpenJDK")
|| vmName.contains("HotSpot")
|| vmName.contains("GraalVM")
|| vmName.contains("Dynamic Code Evolution"),
"Native monitor callbacks require platform-thread classification");
Path recording = Files.createTempFile("preexisting-monitor-callback-", ".jfr");

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 77afe25: assertPreExistingMonitorCallback now starts with assumeTrue(TaskBlockAssertions.platformThreadClassificationSupported()), the guard MonitorTaskBlockTest uses. TaskBlockAssertions and that method are now public so the test in the other package can call it.

@kaahos

kaahos commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Failure while initializing the next JFR chunk can erase the receipt for an already-delivered dump after its source prefix has been truncated. Two new test paths can also fail on valid engine selections or unrelated monitor contention.

Open Bits AI session

🤖 Bits Code Review · Commit 77afe25

}
flush(_buf);
} catch (const OutputFailure&) {
delivery = {};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Preserve completed delivery when the next chunk fails

If initializing the next chunk fails after a successful copy, deliverAndClose() has already closed the destination and restart() has truncated the source. Clearing delivery makes dump report "source bytes retained" and omit certification, although that prefix exists only in the destination and cannot be recovered by retrying the source. Track completed delivery separately from continuation failure while preserving quarantined recording ownership.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and waiting on the author's decision. After deliverAndClose() succeeds, both _output.restart() failing and the OutputFailure catch while writing the next chunk's header return an empty Delivery. dump() then reports "source bytes retained" and skips JFR_DELIVERIES_CERTIFIED, although the delivered prefix now exists only in the destination. Options: (A) return the completed Delivery and let _output.good() == false alone signal the broken continuation, so later dumps fail with "Incomplete recording output is still owned"; (B) add a continuation-failed flag to Delivery, so dump() certifies the delivery and returns a distinct error for the failed next chunk. B is recommended: it keeps the receipt and still tells the caller that the recording can no longer continue. Either option needs a fault-injection test that fails the next chunk's header write after a successful copy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f5e5610: a completed delivery is now certified (JFR_DELIVERIES_CERTIFIED) even when continuing the recording fails, and dump() returns "Recording delivered; continuing the recording failed". The delivered chunk range is released before restart(), so a failed continuation cannot claim it again. Fault-injection gtests cover both the restart() and next-header failure points.

/** Monitor producers classify through JNI, which fails closed on non-HotSpot JDK 19+. */
@Override
protected boolean isPlatformSupported() {
return TaskBlockAssertions.platformThreadClassificationSupported();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Gate producer success tests on wall-engine capability

On OpenJ9 before JDK 19, this predicate accepts runtimes selecting J9WallClock, including dd.profiling.ddprof.j9.sampler=jvmti. That engine lacks unfiltered registry tracking, so Profiler::start() leaves TaskBlock disabled and the positive event assertions fail. Gate these tests on engine capability, as JavaProfilerTaskBlockApiTest does, and apply the same guard to ParkTaskBlockTest.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly fixed in e1a6712: MonitorTaskBlockTest.isPlatformSupported() now returns TaskBlockAssertions.isSupported(), which probes Profiler::taskBlockSupportedForTest (wall-engine registry tracking and JVMSupport::isPlatformThread), the gate JavaProfilerTaskBlockApiTest uses. ParkTaskBlockTest is not changed yet and waits on the author's decision. parkEnter0 does not call isPlatformThread, so reusing isSupported() there would also skip park coverage that works on non-HotSpot JDK 19+. The other option is a probe that checks only the engine, which needs a new native test entry point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4b55c75: ParkTaskBlockTest (and the park case of TaskBlockProducerCompatibilityTest) is gated on a new engine-only probe, TaskBlockAssertions.parkSupported(), which checks wall-engine registry tracking without requiring isPlatformThread, so park coverage keeps running on non-HotSpot JDK 19+.

Comment on lines +53 to +54
for (JfrEvent event : events) {
count++;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Scope delegated-wait assertions to the scenario worker

Monitor-wait delegation leaves native contention producers enabled process-wide. If another JVM thread experiences eligible contention during this recording, its legitimate TaskBlock fails this loop's worker-identity assertions or final count. Restrict counting and validation to the scenario worker while retaining its blocker and state checks.

Suggested change
for (JfrEvent event : events) {
count++;
for (JfrEvent event : events) {
if (!Long.valueOf(workerId).equals(event.getThreadJavaId("eventThread"))) continue;
count++;

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e1a6712: the loop now skips events from other threads and counts only the scenario worker's events. The blocker, state, span and stack checks and the exactly-one count still apply to that worker.

@kaahos

kaahos commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Failure writing the next chunk can discard a completed dump’s delivery receipt and falsely report that its bytes remain in the source. The source has already been truncated, and quarantine prevents retrying delivery.

Open Bits AI session

🤖 Bits Code Review · Commit e1a6712

}
flush(_buf);
} catch (const OutputFailure&) {
delivery = {};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Preserve completed delivery when the next chunk fails

If delivery succeeds but writing the next chunk fails, this catch discards the completed Delivery after restart() has truncated the source. FlightRecorder::dump() consequently reports "source bytes retained" and skips certification, although those chunks exist only in the destination. Quarantine then prevents retrying their delivery. Preserve the completed receipt independently of continuation failure, including the restart() failure branch, and distinguish these outcomes in dump(); this requires coordinated handling beyond this catch.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same finding as the earlier thread; fixed in f5e5610 (completed delivery certified, distinct continuation error, delivered range released before restart()), with fault-injection gtests for both failure points.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

human_review:mandatory Human review required — see PR description test:asan Run CI tests with AddressSanitizer configuration test:fuzz test:release Run CI tests with release configuration test:reliability test:tsan Run CI tests with ThreadSanitizer configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants