Repository navigation
Conversation
CI Test ResultsRun: #44815694480 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled
|
| Test | Cells | Message |
|---|---|---|
UnownedWallTailContextTransitionTest.knownSpanThenClearKeepsOriginalTail__ |
glibc-21-release-amd64 |
org.opentest4j.AssertionFailedError: Coverage incomplete: no retained A tail after span clear ==> expected: <true> but w |
GCGenerationsTest.shouldNotReportPreviousRecordingAdmissions__ |
musl-11-librca-debug-aarch64 |
memory=128,generations=true java.io.IOException: Error occurred while parsing JFR recording |
These fail the build. Passing on a second run makes a test flaky, not passing. Fix it, or quarantine it against a ticket so the debt is tracked rather than forgotten.
Consider quarantining these — click for ready-made entries
A quarantined test still runs and still reports; its failures just stop
turning CI red. To quarantine one:
- Open a PROF ticket for the test, linking the failing job.
- Append the line below to
ddprof-test/quarantine.txt, replacing
PROF-XXXXXwith the ticket number. - Check the
cellsandreasoncolumns — the proposal only knows what
failed in this run, and a narrowercellsglob keeps the same test
gating everywhere it has not misbehaved.
CI fails once review_by passes, so an entry expires instead of piling up.
# test | ticket | added | review_by | cells | reason
com.datadoghq.profiler.wallclock.UnownedWallTailContextTransitionTest.knownSpanThenClearKeepsOriginalTail | PROF-XXXXX | 2026-11-20 | 2027-02-18 | glibc-21-release-amd64 | org.opentest4j.AssertionFailedError: Coverage incomplete: no retained A tail after span clear ==> expected: <true> but was: <false> (seen in: glibc-21-release-amd64)
com.datadoghq.profiler.memleak.GCGenerationsTest.shouldNotReportPreviousRecordingAdmissions | PROF-XXXXX | 2026-11-20 | 2027-02-18 | musl-11-librca-debug-aarch64 | memory=128,generations=true java.io.IOException: Error occurred while parsing JFR recording (seen in: musl-11-librca-debug-aarch64)
Retried 2 of 102 cells.
Failed Jobs
- musl-aarch64/debug / 11-librca — view logs
- glibc-amd64/tsan / 8-ibm — view logs
- glibc-amd64/release / 21 — view logs
Summary: Total: 108 | Passed: 105 | Failed: 3
Updated: 2026-11-20 21:29:30 UTC
1beeda2 to
38efbf8
Compare
Benchmark Results (commit 38efbf8)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125051342 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10365 ms (21 iters) | ✅ 10195 ms (21 iters) | ≈ -1.6% (±11.4%) | — / — |
| akka-uct | 25 | ✅ 8859 ms (24 iters) | ✅ 8925 ms (24 iters) | ≈ +0.7% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5945 ms (33 iters) | ✅ 5953 ms (33 iters) | ≈ +0.1% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5495 ms (36 iters) | ✅ 5431 ms (36 iters) | ≈ -1.2% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2719 ms (68 iters) | ✅ 2830 ms (66 iters) | 🔴 +4.1% | — / — |
| fj-kmeans | 25 | ✅ 2737 ms (68 iters) | ✅ 2815 ms (66 iters) | 🔴 +2.8% | — / — |
| future-genetic | 21 | ✅ 2114 ms (88 iters) | ✅ 2140 ms (87 iters) | ≈ +1.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2080 ms (90 iters) | ✅ 2126 ms (88 iters) | ≈ +2.2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1299 ms (131 iters) | ✅ 1232 ms (139 iters) | ≈ -5.2% (±31.9%) | — / — |
| naive-bayes | 25 | ✅ 997 ms (172 iters) | ✅ 1004 ms (170 iters) | ≈ +0.7% (±31.8%) | — / — |
| reactors | 21 | ✅ 16202 ms (15 iters) | ✅ 15908 ms (15 iters) | ≈ -1.8% (±7.6%) | — / — |
| reactors | 25 | ✅ 18524 ms (15 iters) | ✅ 18484 ms (15 iters) | ≈ -0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2045 / 1958 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 2353 / 2059 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8437 / 8382 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8800 / 8439 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1248 / 1279 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1262 / 1271 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2963 / 2980 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2955 / 2936 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 7 | 3466 / 3539 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 1 | 3478 / 3531 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1700 / 1609 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1873 | ✅ / ✅ | ✅ / ✅ |
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/154349800 |
38efbf8 to
cdb8274
Compare
Benchmark Results (commit cdb8274)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125162043 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| finagle-chirper | 21 | ✅ 5956 ms (33 iters) | ✅ 6031 ms (33 iters) | ≈ +1.3% (±24.8%) | |
| finagle-chirper | 25 | ✅ 5457 ms (36 iters) | ✅ 5444 ms (36 iters) | ≈ -0.2% (±23.9%) | |
| fj-kmeans | 21 | ✅ 2683 ms (70 iters) | ✅ 2821 ms (66 iters) | 🔴 +5.1% | — / — |
| fj-kmeans | 25 | ✅ 2786 ms (67 iters) | ✅ 2839 ms (66 iters) | ≈ +1.9% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2063 ms (90 iters) | ✅ 2095 ms (89 iters) | ≈ +1.6% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2072 ms (90 iters) | ✅ 2125 ms (87 iters) | ≈ +2.6% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1257 ms (135 iters) | ✅ 1250 ms (136 iters) | ≈ -0.6% (±32.8%) | — / — |
| reactors | 21 | ✅ 16137 ms (15 iters) | ✅ 15974 ms (15 iters) | ≈ -1% (±7.5%) | — / — |
| reactors | 25 | ✅ 18395 ms (15 iters) | ✅ 18505 ms (15 iters) | ≈ +0.6% (±4.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2260 / 2182 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8741 / 8673 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 8238 / 8373 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1241 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2971 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2968 / 2956 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3468 / 3474 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 9 / 3 | 3464 / 3465 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1531 / 1536 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1857 / 1856 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit e13ec27)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125211559 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
e13ec27 to
c111812
Compare
Benchmark Results (commit c111812)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125236325 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
c111812 to
df472c0
Compare
Benchmark Results (commit df472c0)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125568035 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | ✅ 8806 ms (24 iters) | ✅ 8924 ms (24 iters) | ≈ +1.3% (±11.1%) | — / — |
| finagle-chirper | 21 | ✅ 6033 ms (33 iters) | ✅ 5987 ms (33 iters) | ≈ -0.8% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5500 ms (36 iters) | ✅ 5495 ms (36 iters) | ≈ -0.1% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2692 ms (70 iters) | ✅ 2827 ms (66 iters) | 🔴 +5% | — / — |
| future-genetic | 21 | ✅ 2075 ms (89 iters) | ✅ 2042 ms (91 iters) | ≈ -1.6% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2036 ms (91 iters) | ✅ 2023 ms (91 iters) | ≈ -0.6% (±2.8%) | — / — |
| naive-bayes | 21 | ✅ 1228 ms (138 iters) | ✅ 1283 ms (133 iters) | ≈ +4.5% (±33.5%) | — / — |
| reactors | 21 | ✅ 16202 ms (15 iters) | ✅ 16354 ms (15 iters) | ≈ +0.9% (±6.6%) | — / — |
| reactors | 25 | ✅ 18212 ms (15 iters) | ✅ 18060 ms (15 iters) | ≈ -0.8% (±4.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 5 | 2176 / 2305 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8602 / 8811 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 8746 / 8149 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1255 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 1280 / 1287 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 2966 / 2985 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 2906 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 3486 / 3512 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3483 / 3521 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1654 / 1756 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1822 / 1798 | ✅ / ✅ | ✅ / ✅ |
df472c0 to
46fdadd
Compare
Benchmark Results (commit 46fdadd)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125595112 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
46fdadd to
658df77
Compare
Benchmark Results (commit 658df77)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125689745 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10260 ms (21 iters) | ✅ 10296 ms (21 iters) | ≈ +0.4% (±10.9%) | — / — |
| akka-uct | 25 | ✅ 8840 ms (24 iters) | ✅ 8942 ms (24 iters) | ≈ +1.2% (±10.6%) | — / — |
| finagle-chirper | 21 | ✅ 6010 ms (33 iters) | ✅ 5968 ms (33 iters) | ≈ -0.7% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5480 ms (36 iters) | ✅ 5467 ms (36 iters) | ≈ -0.2% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2679 ms (69 iters) | ✅ 2773 ms (67 iters) | 🔴 +3.5% | — / — |
| fj-kmeans | 25 | ✅ 2769 ms (68 iters) | ✅ 2839 ms (66 iters) | ≈ +2.5% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2051 ms (90 iters) | ✅ 2133 ms (87 iters) | 🔴 +4% | — / — |
| future-genetic | 25 | ✅ 2047 ms (91 iters) | ✅ 2037 ms (91 iters) | ≈ -0.5% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1254 ms (136 iters) | ✅ 1314 ms (131 iters) | ≈ +4.8% (±33.9%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 983 ms (173 iters) | ≈ -3.2% (±31%) | — / — |
| reactors | 21 | ✅ 16594 ms (15 iters) | ✅ 16990 ms (15 iters) | ≈ +2.4% (±9.1%) | — / — |
| reactors | 25 | ✅ 18557 ms (15 iters) | ✅ 18660 ms (15 iters) | ≈ +0.6% (±4.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1856 / 2016 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2373 / 2253 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 7 | 8704 / 8448 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8503 / 8277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1260 / 1244 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1282 / 1284 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2984 / 2927 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2889 / 2841 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 3476 / 3519 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 3442 / 3468 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1680 / 1865 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1868 / 1860 | ✅ / ✅ | ✅ / ✅ |
658df77 to
746edc2
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds first-class TaskBlock emission for common JVM blocking states (park, Object.wait, monitor contention), and reworks wall-clock precheck/suppression to use lifecycle-owned blocked intervals (instead of the prior once-per-run suppression model) while preserving context-window sampling semantics.
Changes:
- Introduces a new
datadog.TaskBlockJFR event and a public paired Java API (beginTaskBlock/endTaskBlock) plus internal park/monitor producers. - Implements native lifecycle ownership + unfiltered-thread registry tracking and a bounded candidate backfill strategy to prevent O(N) wall-tick regressions.
- Adds broad Java + C++ test coverage (including restart/pre-existing-thread scenarios, JVMTI-delegated variants, and benchmark hooks).
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/WallclockMitigationsCombinedTest.java | Updates combined mitigation test expectations to ensure context-scoped threads remain sampled. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java | New integration tests for owned-block suppression behavior when filter= samples all threads. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java | New tests ensuring unfiltered tracking/registry state is correctly reset across recording restarts. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/TaskBlockAssertions.java | New shared assertions for the datadog.TaskBlock event contract. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java | Refactors precheck tests to the new lifecycle-owned suppression model and unowned blocked weighting counters. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckEfficiencyTest.java | Updates efficiency tests to match lifecycle ownership semantics and explicit unfiltered sampling configuration. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/ParkTaskBlockTest.java | New tests verifying TaskBlock emission from Java-owned park hooks and suppression counters. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/MonitorTaskBlockTest.java | New tests verifying TaskBlock emission from JVMTI monitor callbacks and restart recovery. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java | New subclass to exercise unfiltered precheck via delegated JVMTI stack collection. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedPrecheckTest.java | Updates command strings to use explicit filter= under new behavior. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedParkTaskBlockTest.java | New JVMTI-delegated variant of park TaskBlock tests. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedMonitorTaskBlockTest.java | New JVMTI-delegated variant of monitor TaskBlock tests. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockPreExistingThreadTest.java | New test covering TaskBlock TLS initialization for threads created before profiler startup. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockDisabledTest.java | New test verifying TaskBlock API stays inactive outside any sampling scope. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockApiTest.java | New end-to-end tests for the paired TaskBlock API semantics and rejection rules. |
| ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java | New test ensuring J9 wall sampling doesn’t activate unfiltered precheck tracking. |
| ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerTest.java | Adds child-JVM agentpath initialization tests and pre-existing monitor callback coverage. |
| ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerApiSurfaceTest.java | Updates API-surface assertions: TaskBlock API public, internal hooks stay non-public, monitor delegation APIs are public. |
| ddprof-test/src/test/java/com/datadoghq/profiler/ExternalLauncher.java | Extends launcher scenarios to cover agent compatibility, delegation conflicts, and pre-existing monitor callbacks. |
| ddprof-test/src/test/java/com/datadoghq/profiler/context/OtelContextStorageModeTest.java | Ensures trace context is cleared during cleanup alongside thread context reset. |
| ddprof-test/src/test/java/com/datadoghq/profiler/context/AllNativeContextTest.java | Ensures trace context is cleared during cleanup alongside thread context reset. |
| ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java | Adds beforeProfilerStart() hook and improves teardown to clear trace context reliably. |
| ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java | New helper exposing package-scoped owned-block hooks to JMH benchmarks. |
| ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java | New benchmark measuring wall-clock timer overhead under growing owned-block populations. |
| ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp | Adds capability + filter parsing tests relevant to unfiltered tracking (filter omitted vs filter=). |
| ddprof-lib/src/test/cpp/wallClockCounters_ut.cpp | Updates counter tests to the renamed owned-block suppression metric. |
| ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp | New unit tests for randomized bounded candidate selection/backfill. |
| ddprof-lib/src/test/cpp/taskBlockRecorder_ut.cpp | New unit tests for TaskBlock eligibility, rotation gating, and error accounting. |
| ddprof-lib/src/test/cpp/park_state_ut.cpp | Expands state-machine tests for park and monitor snapshot/exit behavior and owned-block transitions. |
| ddprof-lib/src/test/cpp/jvmSupport_ut.cpp | New unit tests for platform-thread classification (virtual thread rejection) across JNI versions. |
| ddprof-lib/src/test/cpp/frame_ut.cpp | Adds unit test for safe JVMTI->ASGCT frame copying with overlapping buffers. |
| ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java | Adds monitor-delegation-aware getInstance, TaskBlock public API, and thread-parameterized native hooks. |
| ddprof-lib/src/main/cpp/wallClockCounters.h | Renames and redefines wall-clock suppression counter to owned-block semantics. |
| ddprof-lib/src/main/cpp/wallClockCandidateSelector.h | New header implementing bounded randomized candidate visitation/backfill. |
| ddprof-lib/src/main/cpp/wallClock.h | Integrates candidate selector into wall timer loop and adds engine capability declaration. |
| ddprof-lib/src/main/cpp/wallClock.cpp | Reworks precheck logic to rely on owned-block registry suppression and bounded slot lookup backfill. |
| ddprof-lib/src/main/cpp/vmEntry.h | Adds monitor delegation ownership model and initialization result enum. |
| ddprof-lib/src/main/cpp/vmEntry.cpp | Implements JVMTI monitor callbacks producing TaskBlocks, plus one-time bridge ownership negotiation. |
| ddprof-lib/src/main/cpp/threadLocalData.h | Adds per-thread state for park/monitor/taskblock lifecycles and snapshotting. |
| ddprof-lib/src/main/cpp/threadFilter.h | Adds unfiltered registry tracking, TID index, recording epochs, and owned-block suppression candidate checks. |
| ddprof-lib/src/main/cpp/threadFilter.cpp | Implements registry activation, per-recording epoch publication, lock-protected indexing, and slot lookup APIs. |
| ddprof-lib/src/main/cpp/taskBlockRecorder.h | New TaskBlock recorder API + eligibility helpers + rotation-safe activity RAII. |
| ddprof-lib/src/main/cpp/taskBlockRecorder.cpp | Implements TaskBlock duration threshold and recording/cleanup at exit paths. |
| ddprof-lib/src/main/cpp/profiler.h | Adds TaskBlock recording API, rotation gating fields, and bootstrap registration hook. |
| ddprof-lib/src/main/cpp/profiler.cpp | Implements TaskBlock recording through JVMTI stack capture, rotation gating, registry bootstrap for existing threads, and start/stop integration. |
| ddprof-lib/src/main/cpp/jvmThread.h | Adds supportsNativeThreadIdLookup() used for registry bootstrap logic. |
| ddprof-lib/src/main/cpp/jvmThread.cpp | Implements native-thread-id lookup capability check. |
| ddprof-lib/src/main/cpp/jvmSupport.h | Adds platform-thread classification API. |
| ddprof-lib/src/main/cpp/jvmSupport.cpp | Implements platform-thread classification (virtual-thread rejection) using JNI version + IsVirtualThread. |
| ddprof-lib/src/main/cpp/jfrMetadata.h | Adds new JFR type id for datadog.TaskBlock. |
| ddprof-lib/src/main/cpp/jfrMetadata.cpp | Defines datadog.TaskBlock metadata schema. |
| ddprof-lib/src/main/cpp/javaApi.cpp | Wires new JNI entrypoints for TaskBlock API and updates park/block hooks to be thread-aware and emit TaskBlocks. |
| ddprof-lib/src/main/cpp/frames.h | Adds copyJvmtiFrames() helper to safely convert overlapping JVMTI frame buffers. |
| ddprof-lib/src/main/cpp/flightRecorder.h | Adds TaskBlock recording API to the JFR writer interface. |
| ddprof-lib/src/main/cpp/flightRecorder.cpp | Implements TaskBlock event serialization and a rotation-safe record path. |
| ddprof-lib/src/main/cpp/event.h | Adds TaskBlockEvent struct and updates wall-clock epoch suppressed counter naming. |
| ddprof-lib/src/main/cpp/engine.h | Adds wall-engine capability hook supportsUnfilteredWallPrecheck(). |
| ddprof-lib/src/main/cpp/counters.h | Adds new counters for registry/backfill and TaskBlock lifecycle/accounting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark Results (commit 746edc2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125879034 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10315 ms (21 iters) | ✅ 10281 ms (21 iters) | ≈ -0.3% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8869 ms (24 iters) | ✅ 8962 ms (24 iters) | ≈ +1% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6014 ms (33 iters) | ✅ 5992 ms (33 iters) | ≈ -0.4% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5449 ms (36 iters) | ✅ 5424 ms (36 iters) | ≈ -0.5% (±24%) | |
| fj-kmeans | 21 | ✅ 2634 ms (72 iters) | ✅ 2749 ms (68 iters) | 🔴 +4.4% | — / — |
| fj-kmeans | 25 | ✅ 2757 ms (68 iters) | ✅ 2825 ms (66 iters) | ≈ +2.5% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2097 ms (88 iters) | ✅ 2137 ms (87 iters) | ≈ +1.9% (±2.8%) | — / — |
| future-genetic | 25 | ✅ 2067 ms (89 iters) | ✅ 2031 ms (91 iters) | ≈ -1.7% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1253 ms (136 iters) | ✅ 1253 ms (137 iters) | ≈ 0% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1008 ms (169 iters) | ✅ 1002 ms (170 iters) | ≈ -0.6% (±31.9%) | — / — |
| reactors | 21 | ✅ 17619 ms (15 iters) | ✅ 15712 ms (16 iters) | 🟢 -10.8% | — / — |
| reactors | 25 | ✅ 18550 ms (15 iters) | ✅ 18609 ms (15 iters) | ≈ +0.3% (±4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1987 / 1971 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2310 / 2435 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8921 / 8512 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8447 / 8030 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1274 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1280 / 1270 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 2912 / 3053 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2767 / 2834 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 3517 / 3509 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 6 | 3483 / 3497 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1802 / 1666 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1883 / 1920 | ✅ / ✅ | ✅ / ✅ |
1db6b94 to
e9b2308
Compare
This comment has been minimized.
This comment has been minimized.
e9b2308 to
cdbed88
Compare
Benchmark Results (commit cdbed88)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126290898 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10260 ms (21 iters) | ✅ 10222 ms (21 iters) | ≈ -0.4% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8935 ms (24 iters) | ✅ 8930 ms (24 iters) | ≈ -0.1% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 6038 ms (33 iters) | ✅ 5962 ms (33 iters) | ≈ -1.3% (±24.4%) | |
| finagle-chirper | 25 | ✅ 5468 ms (36 iters) | ✅ 5427 ms (36 iters) | ≈ -0.7% (±24%) | |
| fj-kmeans | 21 | ✅ 2764 ms (68 iters) | ✅ 2816 ms (66 iters) | ≈ +1.9% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2802 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2130 ms (87 iters) | ✅ 2084 ms (89 iters) | ≈ -2.2% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2023 ms (91 iters) | ✅ 2105 ms (89 iters) | 🔴 +4.1% | — / — |
| naive-bayes | 21 | ✅ 1254 ms (137 iters) | ✅ 1294 ms (132 iters) | ≈ +3.2% (±32.7%) | — / — |
| naive-bayes | 25 | ✅ 965 ms (176 iters) | ✅ 1011 ms (169 iters) | ≈ +4.8% (±32.6%) | — / — |
| reactors | 21 | ✅ 17025 ms (15 iters) | ✅ 16431 ms (15 iters) | ≈ -3.5% (±7.3%) | — / — |
| reactors | 25 | ✅ 18745 ms (15 iters) | ✅ 18914 ms (15 iters) | ≈ +0.9% (±4.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1951 / 2053 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2293 / 2284 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 8904 / 8718 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8249 / 8246 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1302 / 1261 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1251 / 1245 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 2947 / 2965 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 2754 / 3026 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 8 | 3535 / 3486 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 3472 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1677 / 1597 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1961 / 1906 | ✅ / ✅ | ✅ / ✅ |
cdbed88 to
3208f71
Compare
Benchmark Results (commit 3208f71)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126387908 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
|
@codex review |
…wnership conflict
…monitor callbacks
|
🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you. This run
Stop summary. Commits pushed: 77afe25 (skip the pre-existing monitor tests where platform-thread classification is unsupported) and e1a6712 (gate the monitor tests on engine support, scope the delegated-wait assertions to the worker). Both were fast-forward pushes with no rebase. Threads left for the author: the flightRecorder.cpp:1255 delivery receipt when the next chunk fails (design choice, options posted), and the ParkTaskBlockTest guard (choose ✅ stopped 14:19 UTC · fix-push cap reached; CI running on e1a6712, 2 threads await the author's decision |
|
@DataDog review |
There was a problem hiding this comment.
The pre-existing-thread monitor tests require TaskBlock events on OpenJ9 configurations where native monitor producers intentionally reject every callback, causing test failures on those runtimes.
🤖 Bits Code Review · Commit aec1f67
| private void assertPreExistingMonitorCallback(String target, String expectedState) | ||
| throws Exception { | ||
| Path recording = Files.createTempFile("preexisting-monitor-callback-", ".jfr"); |
There was a problem hiding this comment.
Gate native-monitor success tests on runtime support
On OpenJ9 reporting JNI 19 or later, JVMSupport::isPlatformThread() deliberately returns false, so native monitor callbacks produce no TaskBlock events. Both new pre-existing-thread tests nevertheless require a matching event, causing deterministic test failures on these runtimes. Apply the runtime capability guard used by MonitorTaskBlockTest to their shared helper.
| private void assertPreExistingMonitorCallback(String target, String expectedState) | |
| throws Exception { | |
| Path recording = Files.createTempFile("preexisting-monitor-callback-", ".jfr"); | |
| private void assertPreExistingMonitorCallback(String target, String expectedState) | |
| throws Exception { | |
| String vmName = System.getProperty("java.vm.name", ""); | |
| assumeTrue( | |
| !Platform.isJavaVersionAtLeast(19) | |
| || vmName.contains("OpenJDK") | |
| || vmName.contains("HotSpot") | |
| || vmName.contains("GraalVM") | |
| || vmName.contains("Dynamic Code Evolution"), | |
| "Native monitor callbacks require platform-thread classification"); | |
| Path recording = Files.createTempFile("preexisting-monitor-callback-", ".jfr"); |
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Fixed in 77afe25: assertPreExistingMonitorCallback now starts with assumeTrue(TaskBlockAssertions.platformThreadClassificationSupported()), the guard MonitorTaskBlockTest uses. TaskBlockAssertions and that method are now public so the test in the other package can call it.
…ead classification
|
@DataDog review |
There was a problem hiding this comment.
Failure while initializing the next JFR chunk can erase the receipt for an already-delivered dump after its source prefix has been truncated. Two new test paths can also fail on valid engine selections or unrelated monitor contention.
🤖 Bits Code Review · Commit 77afe25
| } | ||
| flush(_buf); | ||
| } catch (const OutputFailure&) { | ||
| delivery = {}; |
There was a problem hiding this comment.
Preserve completed delivery when the next chunk fails
If initializing the next chunk fails after a successful copy, deliverAndClose() has already closed the destination and restart() has truncated the source. Clearing delivery makes dump report "source bytes retained" and omit certification, although that prefix exists only in the destination and cannot be recovered by retrying the source. Track completed delivery separately from continuation failure while preserving quarantined recording ownership.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Valid, and waiting on the author's decision. After deliverAndClose() succeeds, both _output.restart() failing and the OutputFailure catch while writing the next chunk's header return an empty Delivery. dump() then reports "source bytes retained" and skips JFR_DELIVERIES_CERTIFIED, although the delivered prefix now exists only in the destination. Options: (A) return the completed Delivery and let _output.good() == false alone signal the broken continuation, so later dumps fail with "Incomplete recording output is still owned"; (B) add a continuation-failed flag to Delivery, so dump() certifies the delivery and returns a distinct error for the failed next chunk. B is recommended: it keeps the receipt and still tells the caller that the recording can no longer continue. Either option needs a fault-injection test that fails the next chunk's header write after a successful copy.
There was a problem hiding this comment.
Fixed in f5e5610: a completed delivery is now certified (JFR_DELIVERIES_CERTIFIED) even when continuing the recording fails, and dump() returns "Recording delivered; continuing the recording failed". The delivered chunk range is released before restart(), so a failed continuation cannot claim it again. Fault-injection gtests cover both the restart() and next-header failure points.
| /** Monitor producers classify through JNI, which fails closed on non-HotSpot JDK 19+. */ | ||
| @Override | ||
| protected boolean isPlatformSupported() { | ||
| return TaskBlockAssertions.platformThreadClassificationSupported(); |
There was a problem hiding this comment.
Gate producer success tests on wall-engine capability
On OpenJ9 before JDK 19, this predicate accepts runtimes selecting J9WallClock, including dd.profiling.ddprof.j9.sampler=jvmti. That engine lacks unfiltered registry tracking, so Profiler::start() leaves TaskBlock disabled and the positive event assertions fail. Gate these tests on engine capability, as JavaProfilerTaskBlockApiTest does, and apply the same guard to ParkTaskBlockTest.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Partly fixed in e1a6712: MonitorTaskBlockTest.isPlatformSupported() now returns TaskBlockAssertions.isSupported(), which probes Profiler::taskBlockSupportedForTest (wall-engine registry tracking and JVMSupport::isPlatformThread), the gate JavaProfilerTaskBlockApiTest uses. ParkTaskBlockTest is not changed yet and waits on the author's decision. parkEnter0 does not call isPlatformThread, so reusing isSupported() there would also skip park coverage that works on non-HotSpot JDK 19+. The other option is a probe that checks only the engine, which needs a new native test entry point.
There was a problem hiding this comment.
Fixed in 4b55c75: ParkTaskBlockTest (and the park case of TaskBlockProducerCompatibilityTest) is gated on a new engine-only probe, TaskBlockAssertions.parkSupported(), which checks wall-engine registry tracking without requiring isPlatformThread, so park coverage keeps running on non-HotSpot JDK 19+.
| for (JfrEvent event : events) { | ||
| count++; |
There was a problem hiding this comment.
Scope delegated-wait assertions to the scenario worker
Monitor-wait delegation leaves native contention producers enabled process-wide. If another JVM thread experiences eligible contention during this recording, its legitimate TaskBlock fails this loop's worker-identity assertions or final count. Restrict counting and validation to the scenario worker while retaining its blocker and state checks.
| for (JfrEvent event : events) { | |
| count++; | |
| for (JfrEvent event : events) { | |
| if (!Long.valueOf(workerId).equals(event.getThreadJavaId("eventThread"))) continue; | |
| count++; |
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Fixed in e1a6712: the loop now skips events from other threads and counts only the scenario worker's events. The blocker, state, span and stack checks and the exactly-one count still apply to that worker.
…ated wait to its worker
|
@DataDog review |
There was a problem hiding this comment.
Failure writing the next chunk can discard a completed dump’s delivery receipt and falsely report that its bytes remain in the source. The source has already been truncated, and quarantine prevents retrying delivery.
🤖 Bits Code Review · Commit e1a6712
| } | ||
| flush(_buf); | ||
| } catch (const OutputFailure&) { | ||
| delivery = {}; |
There was a problem hiding this comment.
Preserve completed delivery when the next chunk fails
If delivery succeeds but writing the next chunk fails, this catch discards the completed Delivery after restart() has truncated the source. FlightRecorder::dump() consequently reports "source bytes retained" and skips certification, although those chunks exist only in the destination. Quarantine then prevents retrying their delivery. Preserve the completed receipt independently of continuation failure, including the restart() failure branch, and distinguish these outcomes in dump(); this requires coordinated handling beyond this catch.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Same finding as the earlier thread; fixed in f5e5610 (completed delivery certified, distinct continuation error, delivered range released before restart()), with fault-injection gtests for both failure points.
What does this PR do?:
Adds automatic JVM producers for
datadog.TaskBlockon top of the recorder from #664:SLEEPINGandOBJECT_WAIT.CONTENDED). They also ownObject.wait(WAITING), unless the Java bridge delegates monitor wait to Java instrumentation through the newJavaProfiler.getInstance(libLocation, scratchDir, delegateMonitorWaitEvents)overload.BlockSlotpayloads. A nested or mismatched exit cannot consume another producer's owner.Also changes JFR output handling (PROF-16798): a failed write, copy or close no longer loses the retained recording.
Recording::flush()writes disjoint ranges with boundedpwrite()retries and resets the buffer only after every byte is written.dump()truncates the source only after a complete copy and a successful destination close. A failed copy keeps the sealed chunks, and a later dump delivers them.jfr_output_failures).Motivation:
JVM producers must record one completed interval per blocking call, with the entry context and state, across dumps, context changes and restarts. Dropping a pending unowned tail at ownership would lose wall time that was already observed.
Before this change, a partial copy or a failed close was treated as delivery. The profiler could then truncate or free the only copy of the recording.
Additional Notes:
stop()still tears down all engines and producers. Then it keeps the recording and latches the non-restartableERRORstate; a newstart()is refused. It skipsreleaseRecordingMemory(), so buffers, dictionary roots and uncommitted wall tails stay owned. It still closes the intern gate. On this path pending wall tails are retained, not dropped, andwc_unowned_tail_dropped_at_stopdoes not change. A failed close is not retried, because descriptor ownership after a failed close is platform-dependent.getInstancecall with a different preference throwsIllegalStateException. A native HotSpot wait ends atMonitorWaited, before monitor reacquisition. A delegated Java wait ends at the instrumented return fromObject.wait. A live dump emits no prefix for an open interval; stop/start invalidates open owners.BlockSlotpayloads. Each eligible exit captures a stack synchronously. Monitor blocker hashing runs only after the duration check passes.Thread.isVirtual()(through aMethodHandle, which avoids reflection's varargs allocation). JVMTI monitor callbacks use JNIIsVirtualThread. When that function is missing on HotSpot, the thread is rejected,task_block_virtual_thread_detection_failedis incremented and a one-time warning is logged. On non-HotSpot JDK 19+ the monitor producers record nothing.JfrIntegration.mddocumentsRecordingOutput::Delivery. It certifies native write, copy and close completion only, not consumer decoding or upload.jfr_deliveries_certifiedcounts it.freeChunks()removing poison before unmap. The production fix is main's (PROF-16142).OS::copyFile()uses a fixed buffer instead of mapping the source, and reports partial copies (PROF-16798).build-and-summarizenow writes deterministic summaries tobuild/reports/claude/gradle-summary.{md,json}(utils/gradle_summary.py) and no longer calls an agent (PROF-16796).utils/run-containers-tests.shadds--source-root,--offline,--max-workersand--artifacts-dirfor bounded local runs (PROF-16797).qualify-real-vmGitHub job (debug only): HotSpot 25 and OpenJ9 17 run the producer lifecycle and classifier tests and must pass without skips (.github/scripts/assert_vm_qualification.py). Part of PROF-14931: it guards this PR's producers against tests that skip on a real VM.googleJavaFormat1.30.0 to 1.36.1, with real-formatter TestKit tests inSpotlessConventionPluginTest(PROF-16797).human_review:mandatory). Reason: changesstop()/start()lifecycle on output failure and adds JVMTI callback producers that run on arbitrary JVM threads.How to test the change?:
Added coverage:
recordingOutput_ut(RecordingOutputTest: sticky errors, disjoint reservations, partial write/copy/close),flightRecorder_result_utandflightRecorder_wallEpoch_ut(dump and start output failure branches),profilerStartRollback_ut(ProfilerStartRollbackOutputTest),wallTailRecording_ut(QuarantinedWallTailStopTest),taskBlockProducers_ut,vmEntry_ut(NativeMonitorEventsTest,ProfilerBridgeDelegationTest,MonitorEventConfigurationTest),jvmSupport_ut(JvmSupportThreadClassificationTest),park_state_ut,referenceChainJfrRoundtrip_ut,linearAllocator_nativemem_ut.ParkTaskBlockTest,MonitorTaskBlockTest,JvmtiBasedParkTaskBlockTest,JvmtiBasedMonitorTaskBlockTest,TaskBlockReceivingScopeTest,TaskBlockProducerCompatibilityTest,UnownedTailBeforeTaskBlockTest,BlockingHookAllocationTest,RecordingDeliveryJfrTest,JavaProfilerTest(monitor-delegation singleton conflicts and reuse).HotSpotProducerLifecycleTest,HotSpotDelegatedProducerLifecycleTest,OpenJ9ProducerLifecycleTest,OpenJ9CpuProducerLifecycleTest,RealVmProducerTest,RealVmThreadClassifierTest.utils/test_gradle_summary.py,utils/test_run_containers_tests.py,utils/test_profiler_start_init.py,.github/scripts/tests/test_assert_vm_qualification.py,ProfilerAccessTest.Run:
./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug :ddprof-lib:gtestRelease./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='*TaskBlock*'and-Ptests=RecordingDeliveryJfrTestJAVA_TEST_HOME; thequalify-real-vmjob runs them on HotSpot 25 and OpenJ9 17.For Datadog employees:
dd:platform-security-reviewskill, or file a request via the PSEC review form).bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!