Skip to content

fix(wall): preserve context and sampling-epoch provenance across dumps (PROF-12146) - #890

Open
jbachorik wants to merge 44 commits into
paul.fournillon/wallclock-all-threadsfrom
sphinx/fix-663-review-findings
Open

jbachorik wants to merge 44 commits into
paul.fournillon/wallclock-all-threadsfrom
sphinx/fix-663-review-findings

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Keeps the stack, tracing context and per-epoch weight attached to retained unowned wall-clock tails until a write succeeds, so tails survive call-trace rotation and JFR dumps with their original provenance. Also folds in the follow-ups to #663's review: owner-TLS context-window membership, lock-free thread retirement, timer-join retry, and DEBUG-only test seams.

Motivation:

In filtered precheck mode an unowned blocked thread records about one sample in ten and keeps the rest as pending weight against a cached trace. Before this change the cached trace could be rotated away, the tail was written with the current context instead of the original one, and it was scaled by the epoch in force at flush time rather than at observation time.

Additional Notes:

Human review: mandatory. The change touches signal-reachable sampling, lock ordering (epoch mutex → recording stripes → _rec_lock) and JFR epoch metadata.

Accepted costs (reviewer acknowledgment required):

Cost Why accepted
Closed sampling epochs are kept in Recording::_wall_epochs until a dump or a rare prune (floor 4096 entries, about 400 KiB, charged to wallclock). The prune visits each stripe once as a barrier, never holding all of them. Retained tails need the epoch that was current when they were observed.
Backdated samples and re-emitted epochs can precede a new chunk's start. Consumers scale each sample by the epoch that contains its timestamp.
The block tracker's slots and retained-context chunks are kept until tracker destruction and reported as wall_block_tracker. In-flight owner hooks can carry slot IDs across stop().
A failed timer-thread join keeps the handle; start() retries the join and is refused while it keeps failing. Reusing a live thread handle would be unsafe.
  • Bundled changes outside the PR's purpose are tracked separately: the monotonic wall-timer cadence and DDPROF_WALL_TIMER_DIAGNOSTICS (PROF-16795), the .pi checkout-root fix (PROF-16721), and build/agent tooling changes (PROF-16796, PROF-16797).
  • aarch64 -O0 workaround: WallClockBlockTracker::unownedContextCell() masks the slot as u32. Clang 14 at -O0 on aarch64 lowers the signed masked subscript to a sign-extending byte load, so chunk offsets 128–255 indexed cells −128..−1 (caught by UBSan). UpperHalfChunkOffsetsUseTheirOwnCell fails without it. The comment states when to drop it (clang 15+ floor).
  • Merged with current main through feat(wall): support unfiltered suppression and protect crash bootstrap (PROF-12146) #663: call-trace storage is now recording-scoped, so pending tails are dropped when storage is released at stop.
  • Review: Sphinx standard review of the full range found no Critical or High findings; all Medium findings and the cheap Low findings are fixed with tests. Open Lows: an epoch whose slot root is cleared while segments remain can be pruned mid-recording; a dump can scan the tracker twice inside one critical section.

How to test the change?:

  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug (block tracker, wall epochs, thread filter, call-trace storage).
  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='com.datadoghq.profiler.wallclock.*' and -Ptests='*Lifecycle*'.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-12146

Unsure? Have a question? Request a review!

…me safety

Fixes: mutex acquisition from the pthread TLS destructor (retireThreadByTid
lock-free path), TidIndex use-after-free for in-flight lock-free readers in
~ThreadFilter, capacityExhausted() hint/gate contract, and context-window
state update races.
…cker state fixes

Timer loops re-read recordingEpoch() each tick instead of caching it;
stop() keeps the pthread_t when pthread_join fails; flushUnownedBlockedTail
resets only on valid payloads; owned-run admission preserves unflushed
unowned-blocked weight; test-only exitBlockedRun overload gated behind
UNIT_TEST; epoch publish ordering tightened. Adds lazyBackfill timer-path
coverage and positive-path fuzz assertions for owned-block suppression.
…aProfilerTestSupport to test sources

The registry-activity accessor now returns false in release builds; the
hooks live on JavaProfiler (DEBUG-gated natives) instead of a public
support class shipped in the main jar.
…haos threads

stopProfiler() no longer wipes trace context unconditionally; the
force-wall-start-failure toggle is disarmed in @AfterEach; wall-precheck
counter reads use getOrDefault to avoid NPEs; the parked-worker spin loop
yields; the churn antagonist checks join results and reuses one interruptor
thread.
@jbachorik jbachorik added the AI label Oct 23, 2026
@jbachorik
jbachorik marked this pull request as ready for review October 23, 2026 13:15
@jbachorik
jbachorik requested a review from a team as a code owner October 23, 2026 13:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T13:20:22.326641Z 3f77b78 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f77b7836b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The completed static review identified no reportable defects in deferred thread retirement, wall-clock restart handling, or blocked-time accounting. Runtime behavior was not tested.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 3f77b78

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #44814879809 | Commit: 75e5b98 | Duration: 0s (longest job)

✅ All 0 test jobs passed

No CI outcome reports were found for this run.
Summary: Total: 0 | Passed: 0 | Failed: 0


Updated: 2026-11-20 20:52:51 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 4112b2e0

jbachorik and others added 4 commits November 2, 2026 11:01
Carry the exact SpotlessConventionPlugin content from upstream #955 (e8c3dfb): register Java formatting lazily and use GJF 1.30.0. The stacked #663 base predates that fix; no local formatter variant or formatter-only TestKit is included.
Preserve reusable exact-ID roots and retained context across rotations and recording/context restarts. Keep ordinary sample weight one while accounting for timer work and wake lateness with monotonic deadlines. Add fixed-size phase diagnostics and bounded shutdown/drain regressions without weakening the one-second collapsing oracle.
@dd-octo-sts

dd-octo-sts Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervmwuk1x
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 21.1.8 (6ubuntu1)
Date:Thu Nov 12 23:30:34 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs2
Logic error
Dereference of null pointer1
Unused code
Dead initialization1

Filter Results by File

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Unused codeDead initializationreferenceChains.cppapplyOomRamp4971
Logic errorDereference of null pointerperfEvents_linux.cppunregisterThread72026

@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

❌ 24 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/152683221

❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler gmalloc aarch64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler gmalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler gmalloc amd64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler jemalloc aarch64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler jemalloc aarch64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler jemalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler jemalloc amd64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tcmalloc aarch64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tcmalloc aarch64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tcmalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tcmalloc amd64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer gmalloc aarch64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer gmalloc aarch64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer gmalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer gmalloc amd64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer jemalloc aarch64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer jemalloc aarch64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer jemalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer jemalloc amd64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer tcmalloc aarch64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer tcmalloc aarch64 25 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer tcmalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=1)
❌ chaos: profiler tracer tcmalloc amd64 25 0 3 temXchaos
Chaos harness crashed (RC=1)

Keep unified reservoir accounting and original-context tails. Retain up to
four original-window weight segments and serialize deferred observations
at their original epoch timestamps, committing successful writes separately.
Settle prior sampling statistics before changing pass counts and publish
population boundaries before dispatch.

Consumers must scale by the sample timestamp epoch. Unchanged-pool mid-pass
send failures can still leave coalesced straddling units at the prior success
ratio; that disclosed limitation needs a dedicated follow-up. Lazy sidecar
backing is bounded at 256 KiB. Add scale-aware, wire and retry regressions.
Keep the existing reservoir budgets, pass floors and per-thread assertions.
Publish unique-thread coverage and wait for it within the existing 20-second
timeout instead of assuming a fixed number of random draws covers the pool.

On timeout, report observed passes, coverage and every missing tid before
the fatal assertion, including rejected tids. Add deterministic completion
and fatal-diagnostic regressions without changing the production sampler.
@kaahos
kaahos added this pull request to stack #988 November 12, 2026 23:32
Preserve retained context and epoch tails when skipping clean owned-run clears.
Reconcile shared lazy backing and retain root accounting and generation-checked exits.
Import allocation and Java-8 spin-hint regressions.
kaahos added a commit that referenced this pull request Nov 13, 2026
Keep TaskBlock admission and sampled-generation handshakes on the retained
wall-clock transaction path. Preserve unowned trace provenance and epoch
weights across context changes in filtered and unfiltered stackful sampling.

Retain bootstrap test isolation, bounded sidecar allocation, tooling guards,
and the published TaskBlock review corrections. Keep generations monotonic
on ordinary slot reuse so stale owner tokens cannot clear a replacement run.
kaahos added a commit that referenced this pull request Nov 13, 2026
Skip only pushes without a same-repository open PR head at the pushed SHA. Exercise the workflow shell with hermetic event, head, pagination and error regressions.
@kaahos
kaahos force-pushed the sphinx/fix-663-review-findings branch from 60fac3c to e5c07be Compare November 13, 2026 16:22
@kaahos kaahos changed the title fix(wall): preserve context and sampling-epoch provenance across dumps (PROF-16669) fix(wall): preserve context and sampling-epoch provenance across dumps (PROF-12146) Nov 20, 2026
@kaahos kaahos added the human_review:mandatory Human review required — see PR description label Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI human_review:mandatory Human review required — see PR description test:asan Run CI tests with AddressSanitizer configuration test:fuzz test:release Run CI tests with release configuration test:reliability test:tsan Run CI tests with ThreadSanitizer configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants