Repository navigation
Conversation
CI Test ResultsRun: #44817218192 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled
|
| Test | Cells | Message |
|---|---|---|
NettyEpollIsolationTest.survivingOwnerAndNewWorkerAreDisjointAcrossRestart__ |
glibc-25-graal-debug-amd64 |
org.opentest4j.AssertionFailedError: isolated Netty test failed; retained log=/home/runner/work/java-profiler/java-profi |
NettyEpollIsolationTest.defaultEventLoopWaitProducesTaskBlock__ |
glibc-8-ibm-debug-amd64 |
java.lang.AssertionError: isolated Netty JVM exceeded outer deadline; log=build/netty-epoll/defaultEventLoopWaitProduces |
These fail the build. Passing on a second run makes a test flaky, not passing. Fix it, or quarantine it against a ticket so the debt is tracked rather than forgotten.
Consider quarantining these — click for ready-made entries
A quarantined test still runs and still reports; its failures just stop
turning CI red. To quarantine one:
- Open a PROF ticket for the test, linking the failing job.
- Append the line below to
ddprof-test/quarantine.txt, replacing
PROF-XXXXXwith the ticket number. - Check the
cellsandreasoncolumns — the proposal only knows what
failed in this run, and a narrowercellsglob keeps the same test
gating everywhere it has not misbehaved.
CI fails once review_by passes, so an entry expires instead of piling up.
# test | ticket | added | review_by | cells | reason
com.datadoghq.profiler.wallclock.NettyEpollIsolationTest.survivingOwnerAndNewWorkerAreDisjointAcrossRestart | PROF-XXXXX | 2026-11-20 | 2027-02-18 | glibc-25-graal-debug-amd64 | org.opentest4j.AssertionFailedError: isolated Netty test failed; retained log=/home/runner/work/java-profiler/java-profiler/ddprof-test/build/netty-epoll/survivingOwnerAndNewWorkerAreDisjointAcrossRes (seen in: glibc-25-graal-debug-amd64)
com.datadoghq.profiler.wallclock.NettyEpollIsolationTest.defaultEventLoopWaitProducesTaskBlock | PROF-XXXXX | 2026-11-20 | 2027-02-18 | glibc-8-ibm-debug-amd64 | java.lang.AssertionError: isolated Netty JVM exceeded outer deadline; log=build/netty-epoll/defaultEventLoopWaitProducesTaskBlock-4211798787592456365/child.log (seen in: glibc-8-ibm-debug-amd64)
Retried 2 of 32 cells.
Failed Jobs
- musl-amd64/debug / 8-librca — view logs
- musl-amd64/debug / 25-librca — view logs
- musl-amd64/debug / 21-librca — view logs
- musl-amd64/debug / 11-librca — view logs
- glibc-amd64/debug / 8-ibm — view logs
- musl-amd64/debug / 17-librca — view logs
- glibc-amd64/debug / 25-graal — view logs
Summary: Total: 32 | Passed: 25 | Failed: 7
Updated: 2026-11-20 21:42:50 UTC
9de97d6 to
ddd6385
Compare
This comment has been minimized.
This comment has been minimized.
Scan-Build Report
Bug Summary
Reports
|
||||||||||||||||||||||||||||||||||||
ddd6385 to
cdf93b0
Compare
cdf93b0 to
003c443
Compare
003c443 to
1347a00
Compare
1347a00 to
54c2392
Compare
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/154352147 |
54c2392 to
0bd231e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0bd231e185
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…le-generation guards
|
🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you. This run
✅ stopped 15:21 UTC · base advanced mid-run; CI red (IBM errno tests in parent layer, musl cache post-step); Bits FAIL with 2 P2 open |
|
@DataDog review |
There was a problem hiding this comment.
Netty qualification can accept missing intervals because recorder-rejection credit is reused across workers. The isolated JVM launchers also omit the ASan heap-placement settings used by the parent tests.
🤖 Bits Code Review · Commit ab25d4c
| Path run = Files.createTempDirectory(directory, method + "-"); | ||
| List<String> command = new ArrayList<>(); | ||
| command.add(Paths.get(System.getProperty("java.home"), "bin", "java").toString()); | ||
| command.add("-Xmx512m"); |
There was a problem hiding this comment.
Preserve ASan JVM settings in isolated child commands
On Linux ASan runs with JDK 12+, the child inherits LD_PRELOAD and ASAN_OPTIONS but omits the parent's HeapBaseMinAddress and CompressedClassSpaceSize settings. ConfigurationPresets.configureAsan() supplies these specifically to prevent JVM heap placement from colliding with ASan shadow memory on affected kernels. The isolated tests can therefore fail during JVM startup before exercising Netty. Preserve the required sanitizer JVM settings in both this launcher and TimedNativeWorkerTest.childCommand(), which has the same omission.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Fixed in 0965577: child JVMs now inherit the parent's -Xmx, -XX:HeapBaseMinAddress= and -XX:CompressedClassSpaceSize= through NettyEpollJvm.heapLayoutArgs() (last occurrence wins; -Xmx512m only when the parent sets none). Used by NettyEpollIsolationTest and TimedNativeWorkerTest.childCommand(); covered by NettyEpollJvmArgsTest and an ASan run of NettyEpollIsolationTest.
| } | ||
| stopProfiler(); | ||
| JfrEvents events = verifyEvents("datadog.TaskBlock", false); | ||
| assertEpollTaskBlock(events, name, "io.netty.", recordFailures() - failuresBefore); |
There was a problem hiding this comment.
Avoid crediting one recorder rejection to both workers
Both workers' assertions receive the same recording-wide recorder-failure delta. Four ordinary events, four timed events, and one recorder rejection satisfy both five-interval assertions, although two missing intervals require explanation. This lets lost Netty intervals pass qualification. The shaded and restart cases repeat the pattern. Use separate measurement windows or consume a shared rejection budget across all workload assertions so each rejection explains at most one missing interval.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Fixed in 302370d: both workers now draw on one rejection budget. TaskBlockRecorderTestSupport.assertCompletedIntervals(recordFailures, worker(...)...) requires the summed shortfall to fit the single delta, and a surplus on one worker cannot offset another's shortfall. Applied to the default loop, shaded image, restart and survivor cases; covered by TaskBlockRecorderCountAssertionsTest.
What does this PR do?:
Extends the shared native I/O patches from #666 to Netty's native epoll transport library, including shaded copies outside the JDK directory.
epoll_pwait2handling with the same errno-preserving scope as the other epoll waits.ThreadFilter::refreshOwnerSlotForRecording()validates a retained native owner (TID, lifecycle generation, recording epoch) under the registry lock before it provisions or resets recording payload.Motivation:
Netty epoll calls libc directly and does not go through the JDK networking imports, so #666 does not see its waits. Shading and deleted extraction files defeat path-based discovery. Patching by file name, or scanning arbitrary mapped symbols, could patch an unrelated library.
A Netty worker often blocks across a recording restart. Its retained owner must get the new recording's projection without clearing a different thread's native lifecycle.
Additional Notes:
epoll_pwait2on older glibc: when libc has noepoll_pwait2(glibc before 2.35), everyepoll_pwait2import stays untouched, including a non-NULL lazy-binding stub. A NULL weak import stays NULL, so Netty's availability probe keeps its meaning. Neither case disables the hooks or skips the image.native_io_netty_images_skipped. Other libraries keep their patches and both owners stay active. Why: one unusual Netty build must not disable all socket hooks. The skip is keyed by base and fingerprint, so other cached descriptions of the same image are also skipped. The next installation (normally the next recording) retries it once.io.netty:netty-transport-classes-epollandnetty-transport-native-epoll4.1.137.Final (x86_64 and aarch_64) intestCommon. No production dependency. The real-Netty fixture qualifies Linux glibc only, because the published Netty native artifacts need glibc; musl keeps the profiler's own epoll, weak-import and lifecycle tests.AbstractProfilerStopRetryTest); terminal errors still fail the test..github/scripts/prepare_reports.shcopiesddprof-test/build/netty-epoll(child JVM argv, logs,hs_errfiles and JUnit XML) intotest-reports/, with a case intest_prepare_reports.sh.human_review:mandatory). Reason: patches GOT entries of third-party (Netty) DSOs outside the JDK, and changes failure handling from fail-closed to per-image skip.How to test the change?:
Added coverage:
nettyEpollDiscovery_ut(NettyEpollDiscoveryTest,NettyEpollAliasDiscoveryTestfor removed symlinks and deleted backing files,NettyEpollImageSkipTestfor each skip reason, skip-table capacity, pin release, writable GOT, stale generation and missing libcepoll_pwait2),LibraryPatcherSocketStateTest,NativeSocketInterposerHookTest,NativeBlockScopeTest,nativeIoTimeout_ut,threadFilter_ut(ThreadRegistryTestretained-owner refresh).NettyEpollTaskBlockTest(positive-timeout waits on a verifiedEpollEventLoopworker, default select strategy, JFR fields and stacks),NettyEpollIsolationTest,NettyEpollSyscallAbiTest,ProfilerEpollFixtureTest,NettyEpollTaskBlockAssertionsTest,TimedNativeWorkerTest,NativeWaitTestSupportTest,AbstractProfilerStopRetryTest,TaskBlockRecorderCountAssertionsTest,JavaProfilerTaskBlockLightweightTest..github/scripts/tests/test_prepare_reports.sh.Run:
./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug_nettyEpollDiscovery_ut./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='*NettyEpoll*'on Linux glibcbash .github/scripts/tests/test_prepare_reports.shFor Datadog employees:
dd:platform-security-reviewskill, or file a request via the PSEC review form).bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!