Skip to content

Fix reference-chain nightly flakes, cap the anchor index, and drop their quarantine - #899

Open
jbachorik wants to merge 47 commits into
mainfrom
fix-nightlies
Open

jbachorik wants to merge 47 commits into
mainfrom
fix-nightlies

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Fixes reference-chain discovery and correlation failures reported in PROF-16305, removes the four quarantines added by #888, bounds anchor-index growth under sustained root churn, and removes the one stale gtest quarantine entry (LineNumberTableCopyTest.RejectsUnmappedSource, a test that no longer exists) It also integrates the merged rework of #889 (UNKNOWN_TRUNCATED shape handling, the 0→resolved class-tag upgrade) and addresses the two pre-release quarantine candidates from PROF-16825: both ExternalProcessReferenceChainTest.shouldReconstructReferrerChainForGrowingStaticFieldCollection (chain-found) and LeakTagCorrelationReferenceChainTest.shouldRotateFrontierLeakHoldersUnderPodLikeTightWindows pass on this branch; full-matrix confirmation runs in CI.

  • Post-merge fix push (e0fd8a5): the five actionable datadog-code-review findings on the merged delta are fixed (fair-tier picks hand their queued first look to the caller's requeue; saturated passes keep non-indexed/evicted queue entries pending; promotion preserves displaced incumbents' epoch samples; heapRootCallback journals DURABLE_ADD on fresh durable admission; stale join-count assertion removed), plus three follow-up replies naming deferred items (per-source retry stamps, flat-over-bar takeover semantics, shape-reconciliation seeding). The full-range Sphinx review of the pushed head returned 0 Critical/High; 8 Low + 1 Info observations are triaged as documented bounds or follow-up candidates. The manually dispatched nightly slow-matrix run passed on rerun across all 118 cells, covering both PROF-16825 quarantined-candidate tests.

  • Resolved-chain cache (PROF-16316): distinguish eviction from a changed-but-live path; rebuild changed paths from the intact frontier and retain the old chain if rebuilding fails.

  • Discovery and rotation: register candidates before the first walk, improve candidate admission, re-enlist evicted durable anchors, retain rotation eligibility for reparented collections/maps, and reclaim collected local anchors.

  • Callback-safe indexing: heap callbacks record anchor-index observations in a fixed 16,384-record journal. The traversal caller replays accepted observations after each synchronous JVMTI call returns. On journal overflow, the walk aborts before the associated frontier transition, replays the accepted prefix, and retries the unserved work.

  • Retry progress: root-enumeration retry progress survives independently of the capped durable-anchor index. The frontier table stamps the last attempt that accepted each durable-root report by source, kind, and attempt; a retry skips restaging only those same-kind accepted reports, so repeated journal overflows cannot trap enumeration at the same prefix and starve later roots.

  • Liveness thread trends: the challenger-thread trend scratch is extracted to its own unit and retains four challenger slots per class (143,360 B fixed payload across 256 classes), so a rising allocating thread cannot be evicted while the incumbent cohort list stays at 16 slots.

  • Lifecycle and ordering: a failed BFS-thread join prevents tracker teardown; at-risk observations return to the queue in their original FIFO order.

  • Slow-test pacing: give the CPU-budget-paced walk time to drain its backlog without weakening assertions, retries, or configured limits.

  • Test hygiene: the ELF parser's unmap test skips under TSan in-test (PROF-16632) instead of relying on a quarantine entry.

Motivation:
The 2026-10-23 Nightly Sanitized run 39893765385 had 20 failed slow-test jobs. One musl-x64 JDK 11 failure reported the ThreadLocal and leak-tag correlation tests failing, and no CachedPayload chain after 69,001 cache entries. #888 quarantined these slow tests.

The late ThreadLocal failure exposed a cache-lifecycle gap: dump-time invalidation erased a changed chain, while a terminal search could clear the candidate state needed to rebuild it. The fix rebuilds path-changed chains from the still-intact frontier. Candidate registration now also precedes the first walk, so a fast-draining pass cannot finish before candidate-specific discovery is active.

The cache fixture now grows to at most 36,601 entries (down from 69,001), then leaves polling rounds for the paced walk to catch up. Assertions and configured limits remain. The anchor-index fix addresses the cost of re-enlisting evicted anchors: before the fix, the rotation test selected the same 32 holders over 20 passes, leaving 8,968 of 9,000 unselected.

Additional Notes:

Accepted costs — reviewer acknowledgment requested:

  • The durable index caps at 8,192 and FIFO-evicts in batches of 1,024 down to 7,168; the local index caps at 4,096. If the local share is full of live collections/maps, new local holders wait until entries die or the search restarts. Selection and shape reconciliation scan up to 12,288 entries per pass; this PR leaves that recurring cost unchanged for follow-up.

  • Each pass selects up to 32 indexed anchors and drains up to 16 at-risk FIFO anchors; actual walks also obey edge and deadline budgets. For 9,000 durable holders, 282 passes is only the ideal minimum with 32 unique selections every pass; rotation is best-effort and can take longer.

  • Staging skips records whose replay is a known no-op (re-enlist of an indexed tag; LOCAL of an indexed/parked/classified-non-container holder; LOCAL of a confirmed container refused by a full pending buffer with no displaceable victim; AT_RISK of an already-queued, cap-blocked, or quota-blocked holder): a stable anchor population above the journal capacity no longer refills the journal with its own no-op records every enumeration, so the overflow can no longer abort at the same prefix every pass and starve every later root.

  • A retry after an incomplete root enumeration suppresses restaging only for reports the preceding incomplete attempt (or the same attempt earlier) accepted with the same root kind; rejected transitions and root-kind changes still restage. Each durable-root report pays one shared-lock frontier probe per enumeration attempt; the anchor-index check is a lock-free set probe.

  • LOCAL reports beyond what replay can absorb (local-index room, pending-buffer slots, displaceable victims) are not staged at all; those holders are re-offered by the next enumeration and admitted when capacity frees. A refused LOCAL population therefore cannot fill the journal. A fixed per-tracker counter tracks displaceable victims without rescanning the pending buffer.

  • The journal occupies up to 384 KiB per tracker, plus 136 KiB of per-scope dedup indexes. On staging overflow, the affected walk can truncate and retry. Each accepted observation currently takes separate journal-lock acquisitions for reservation and commit; that overhead is not optimized here.

  • An eviction requests a static-field sweep without resetting the active cursor. The sweep processes at most 512 loaded classes per pass; requests arriving during a lap remain unserved until a later complete lap. Sustained eviction can therefore add loaded-class sweep work. Resweep pacing is unchanged; representative above-cap measurement remains follow-up.

  • #898 and #889 remain open and overlap this change. This PR does not include all of their current follow-up changes; reconcile the combined tree before merging.

  • Quarantine flaky reference-chain slow e2e tests (PROF-16305) #888 is merged; this PR removes its four PROF-16305 quarantine entries.

  • Agentic review: two Sphinx review rounds. The first found six LOW findings; commits 383649e03, c2afa4c89, and 5d9fb5381 fix all six (lease-checked journal snapshot, shape-first LOCAL lookups, local-subset regression test, and rotation boundary tests). The re-review verified those fixes and found three new LOW performance observations, none waived: the per-pass full-index rotation scan (removal is the planned-but-unapproved maintained tier-index follow-up), per-candidate staging lock/hash lookups in the journal callbacks, and the worst-case-sized per-scope at-risk staging table. No Critical/High findings remain. Separate Bits follow-ups remain tracked outside this PR.

  • Human review is recommended for the fixed journal footprint and unchanged resweep policy.

How to test the change?:

  • :ddprof-lib:gtestDebug_referenceChains_ut: 350 passed. Full :ddprof-lib:gtestDebug: 1,413 passed, 11 skipped. buildRelease succeeded.
  • :ddprof-test:testDebug -Ptests=*ReferenceChain*: 22 passed. :ddprof-test:testSlowDebug -Ptests=*ReferenceChain*: 10 passed, including the late static-field append case.
  • LivenessTrackingTest and ProfilerLifecycleTest.testLivenessRestartCycles pass.
  • qualityChecks, spotlessCheck, both quarantine validators, and the copyright/CLAUDE.md checks pass. Native builds on this macOS host used Xcode 26.4's SDK because the default CommandLineTools 27.2 SDK linker rejected arm64e.x1-* TAPI entries.
  • The static-field E2E assertion is unchanged. Debug builds drive at most eight pass-seam attempts after the late append to test re-expansion without depending on organic scheduler cadence; release builds keep the organic cadence.
  • Regression coverage includes journal capacity/overflow/retry, lease-checked snapshot, deferred callback mutations, FIFO requeue, failed-join teardown, anchor re-enlistment, local-share subset preservation, completed-lap resweeps, fresh-lane budget boundaries, liveness challenger retention, and root-enumeration retry progress.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16305, PROF-16316, PROF-16632, and PROF-16825.

Unsure? Have a question? Request a review!

@jbachorik jbachorik added the AI label Oct 25, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #45074431349 | Commit: bf090aa | Duration: 22m 30s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-22 22:18:00 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

❌ 38 passed, 0 failed, 2 without a result out of 40 expected configurations

Test Matrix

Platform JDK 8 JDK 11 JDK 17 JDK 21 JDK 25
glibc-x64-hotspot ❔ ✅ ✅ ✅ ✅
glibc-x64-openj9 ❔ ✅ ✅ ✅ ✅
glibc-arm64-hotspot ✅ ✅ ✅ ✅ ✅
glibc-arm64-openj9 ✅ ✅ ✅ ✅ ✅
musl-x64-hotspot ✅ ✅ ✅ ✅ ✅
musl-x64-openj9 ✅ ✅ ✅ ✅ ✅
musl-arm64-hotspot ✅ ✅ ✅ ✅ ✅
musl-arm64-openj9 ✅ ✅ ✅ ✅ ✅

No result produced

These configurations wrote no validation log, so their outcome is unknown and
they are counted as failures. The job usually timed out or failed during setup;
see its log in the pipeline.

  • glibc-x64-hotspot-jdk8
  • glibc-x64-openj9-jdk8

Links

@jbachorik
jbachorik marked this pull request as ready for review October 28, 2026 14:56
@jbachorik
jbachorik requested a review from a team as a code owner October 28, 2026 14:56
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T14:59:42.438407Z 0ce265c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ce265c9ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jbachorik jbachorik changed the title Fix reference-chain nightly flakes (PROF-16305) and drop their quarantine Fix reference-chain nightly flakes and drop their quarantine Oct 29, 2026
jbachorik added a commit that referenced this pull request Nov 2, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/152672907

jbachorik added a commit that referenced this pull request Nov 3, 2026
@jbachorik

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/main/cpp/referenceChainAnchors.cpp:437-458 — compactLocalAnchors() now iterates only _local_anchor_tags (now a tag->own-class-tag map, referenceChains.h), so the locked scan is O(cap) and the per-slot count() double hash is gone; referenceChainAnchors.cpp:413-423 — a compaction that removes nothing arms LOCAL_ANCHOR_COMPACT_BACKOFF=8 skipped attempts, so a saturated-but-live share no longer pays the scan every pass (pinned by LocalRootAnchorCompactBackoffSkipsNoProgressCompactions). The O(index) vector compaction remains but runs only when entries are actually removed.

@jbachorik

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/main/cpp/referenceChainAnchors.cpp:362-392 — at a full pending buffer a confirmed container (met at the local cap) evicts the oldest not-yet-confirmed entry via the forward-only _pending_local_evict_cursor, dedup set kept exact; it is dropped only when every parked entry is already a confirmed container (pinned by LocalRootAnchorPendingBufferEvictsForConfirmedContainer).

@jbachorik

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/main/cpp/referenceChainAnchors.cpp:641-667 — collection step extracted verbatim into const collectUnclassifiedAnchorClassTags(); ddprof-lib/src/test/cpp/referenceChainsAnchorTests.inc:1167 AnchorShapeReconcileBudgetCapsRequestedClasses parks budget+10 distinct classes and asserts the request is exactly ANCHOR_SHAPE_RECONCILE_BUDGET in order (fails for <= or a removed stop) and that the unrequested remainder is dropped by resolvePendingLocalAnchorsForTest().

@jbachorik

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/test/cpp/referenceChainsAnchorTests.inc:1209 AnchorShapeReconcilePrefersPendingHolders — phase 1 asserts a full budget of pending classes is requested with no durable class (fails if loop order is swapped, or if room && is dropped from the index loop since the size would exceed the budget); phase 2 asserts pending classes first, then durable classes in index order, size == budget.

@dd-octo-sts

dd-octo-sts Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervmccnms
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 21.1.8 (6ubuntu1)
Date:Wed Nov 18 11:04:54 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs2
Logic error
Dereference of null pointer1
Unused code
Dead initialization1

Filter Results by File

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Unused codeDead initializationreferenceChains.cppapplyOomRamp5321
Logic errorDereference of null pointerperfEvents_linux.cppunregisterThread72026

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Durable-anchor eviction lacks a working traversal-based readmission path. Once the index exceeds its cap, live collections can lose targeted walks, causing later backing-storage replacements and leak instances to be missed.

Open Bits AI session

🤖 Bits Code Review · Commit 42f5c0b

@datadog-prod-us1-4

This comment has been minimized.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR preserves leak chains, improves traversal/admission, hardens slow tests, and removes their quarantine; the latest commit restores callback-based anchor readmission. The earlier finding remains partially open: sweep completion can discard resweep requests, and eviction-driven cooldowns can stall, leaving static-only collections unindexed and missing later leak instances.

Open Bits AI session

🤖 Bits Code Review · Commit 6444508

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR preserves leak chains, improves traversal, hardens slow tests, and removes quarantine; callback-based anchor readmission is fixed. The earlier scheduling finding remains open despite the pending flag: sweep completion still loses requests and cooldown suppresses later requests, so static-only holders can miss targeted leak discovery.

Open Bits AI session

🤖 Bits Code Review · Commit 957ad12

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR preserves leak chains, improves traversal, hardens slow tests, and removes quarantine; earlier request-loss and cooldown mechanisms are fixed. Forced resweeps can still stop after one chunk, excluding later static-only holders from targeted leak discovery, and the new cooldown regression test never triggers an eviction.

Open Bits AI session

🤖 Bits Code Review · Commit 85eab40

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR preserves resolved leak chains, improves traversal/admission, hardens slow tests, and removes quarantine. Durable-anchor resweeps now remain enabled across chunks until completion, and the cooldown regression case refills to capacity and checks an eviction-generation increment; both reported findings are resolved, with no open findings remaining.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 23f8861

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Repeated durable-anchor eviction can confine the static-field sweep to its first 4,096 classes. Reparented collections can also lose persistent rotation eligibility after eviction, impairing discovery of replacement backing storage.

Open Bits AI session

🤖 Bits Code Review · Commit 5da1b2c

A stable anchor population above the journal capacity re-staged the same
no-op LOCAL/REENLIST records every enumeration; the journal overflowed at
the same prefix, the enumeration aborted there every pass, and no later
root was ever reached. Stage only when the replay would change state:
skip REENLIST for already-indexed tags and LOCAL for indexed, parked or
classified-non-container holders.
A retry of the first static-field chunk re-latched the current resweep
generation, but its retry stamps suppress re-reports of holders the
failed attempt already accepted. A request raised between the original
latch and the retry - an at-cap eviction while replaying the accepted
prefix, for one - was then published as served by a lap that never
re-indexed the evicted holder, leaving a static-only holder out of the
index with no pending resweep. Keep the original latch on retry; the
next fresh lap from class 0 serves the newer generation.

(cherry picked from commit 7ed670f8ad7f7d46328121a28546748870a8e79e)
A newcomer with the incumbent table full always accrues in the challenger
pool instead of immediately evicting a sub-bar incumbent: two alternating
single-sample threads could otherwise overwrite the same slot every epoch
and never reach the retained-count bar (4631990044). Challenger promotion
remains the only path back into the table and still requires outranking
the weakest incumbent. Adds the newcomer-churn bound, age-capacity guard,
strict-promotion and consecutive_positive-reset test coverage.

(cherry picked from commit 2c4cdd8d4333f56373ca064a08dd83fc88563efa)
Drops static_report_seen from the weaker-JNI suppression disjunction so a
holder whose static edge disappeared can re-enlist through its JNI global
after eviction (4631990008), and widens the incomplete-retry stamp window
to the whole retry sequence so alternating oversized JNI groups cannot
ping-pong each other out of the durable index (4632610528). The widened
window expires when an attempt completes. Updates the suppression doc
wording and the stamp-window tests.

(cherry picked from commit 3ab9ac0e28a4d401c6819ef9c28122543a3ec313)
Selection no longer marks a fresh anchor's first look served: walked
anchors are served post-descent (after the scope flush, so the walk's own
replay cannot re-enqueue them) and unwalked fresh picks are requeued to
the fresh-queue front with their own_class_tag threaded to the caller
(4631990027). When the leak tier saturates the selection budget the fresh
lane gets no budget, so queued entries spend their first looks then and
fall back to the fair tiers (F-b7bd3229adbc). A journal overflow mid-
descent rolls the partially walked anchor's edges back out of the budget
(F-433ed831a6bc), and the at-risk staging cap check now runs before the
class-set slot claim so a rejected class cannot burn a slot
(F-6ebf4cd70b59).

(cherry picked from commit 5ca4a3c1ef23c05db478785c0caccb47f5ccbfcd)
stop() flips _reference_chains_active before stopThread() so no heap
callback or thread-registration path past the flip stages observations
for a consumer that is about to stop existing (F-45af4d770829); a failed
join reopens the gate so the retry re-enters the same path, and
stopThread() stays idempotent for that retry (join stub covered by the
existing failed-join test, now also asserting the gate is closed at join
time).

(cherry picked from commit 7319b2284ca18c2dfc11b2c54a516e11afdb28ac)
(cherry picked from commit 4628efc4e4e62d1c39ebc1d46f6979b46a916012)
…ounted

The leak tier's max_picks cap skips the scan-order suffix of eligible
leak-tagged anchors; since selection no longer spends first looks, a
skipped pick that still holds a fresh-lane turn could never spend it on a
walk (leak-tagged entries never fall to the fair tiers and the fresh
lane's filter rejects them) - the scan now spends the suffix's first
looks and drops them from the queue, keeping leak-tier priority semantics
and the cap as the only wait (F-092d2bc8b1b6). A journal overflow
mid-descent no longer rolls the partial descent's edges back out of the
budget: ctx.edges_admitted counts NEW frontier admissions only, the
accepted prefix stays applied (the caller's flush replays it), and the
re-walk re-charges only still-unadmitted edges - keeping the partial
edges counted is what makes the segment's total charge exact
(F-c2bec0949595). The fresh-kept membership set replaces the
O(picks x kept) vector+find probe (F-9019cf62774a).

(cherry picked from commit 45a3936c84156c2472d658604db9a2daf1b17acc)
A full trend table evicts a slot for a newcomer only when the victim is
both unsustained and dead-enough: two or more trailing zero epochs (the
thread is really gone), or a newest live retained count below the
newcomer's current epoch count. A trend whose consecutive_positive was
just reset by one skipped epoch keeps its slot against a weaker newcomer,
which is dropped and retries on a later epoch - bounded, no new state
(F-772fa299c1da). Adds the takeover evidence tests, the >= boundary case
for the age-capacity guard (F-4116f9e0100c) and the fold-slot bounds
guard test (F-fa7b4443cec4).

(cherry picked from commit 429e441c5a38c2ef45b9e182b05cf94c71ebefb1)
Three fixes from the sealed review of the first-look round: the fair tiers
probe fresh-kept membership with the set's count() instead of a linear
std::find per pick; the all-selections-empty return requeues the fresh
picks popped by selection instead of stranding them out of the queue; and
walkCandidateThreadLocals() counts only fully walked anchors on observation
overflow, matching walkStaticFieldAnchors(). The reference-chain doc states
the compaction cost driver (eraseAnchorEntries' full-index stable
compaction), the self-limiting forced-lap bound, and the release's
root-enumeration pause expectation.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Two of the most critical issues involve lost progress state: in referenceChainFrontier.cpp, static sweeps overwrite accepted JNI-global stamps for root-attached holders, causing journal exhaustion and root starvation; in referenceChainAnchors.cpp, fair-tier-selected holders have their queue entries erased without rollback coverage, permanently stranding FIRST_LOOK_PENDING without a queued turn.

Open Bits AI session

🤖 Bits Code Review · Commit f8fb5c3

Merge resolution notes:

Ported from the reworked PR #889 (0736be0):
- UNKNOWN_TRUNCATED AnchorClassShape: a bounded class-hierarchy walk that ended
  without proving a non-match is no longer cached as NON_CONTAINER. Accepted in
  every shape consumer: isSelectableAnchor, reserveLocalStaging,
  resolvePendingLocalAnchors, localHolderNeedsObservation, noteLocalRootHolder,
  the pending-local eviction-cursor protection, the fair-tier container
  classification, the fresh-lane keep check, and the local index/pending budget
  gate in referenceChainInternal.h. classImplementsContainerOrMap() returns the
  shape and reconcileAnchorClassShapes() caches it (auto-merged).
- The 0 -> resolved class-tag upgrade on repeated anchor sightings, via the new
  _static_anchor_index_class_tags map of StaticAnchorIndexEntry{class_tag, pos},
  maintained alongside the parallel _static_anchor_own_class_tags vector
  (indexAnchorForRotation() dedup-upgrades in place; eraseAnchorEntries()
  rebuilds positions after its stable compaction). The upgrade is triggered
  through this branch's journal: replayAnchorObservations() now calls
  indexAnchorForRotation() for every REENLIST_ELIGIBLE record (its dedup makes
  it a no-op except for the upgrade), and heapReferenceCallback()'s static-edge
  re-enlist no longer excludes already-indexed holders.
- Seeded-class-history liveness fixture support (test_seeded_history) for the
  SeededClassHistory* tests.
- pollWatchedTargets() before the first walk was already present on this branch;
  nothing to port there.

Kept from this branch (f8fb5c3 side): the bounded anchor journal (heap
callbacks only record observations; scope.flush()/replayAnchorObservations()
applies them after the JVMTI call returns), the pending-local buffer with its
eviction cursor, the fresh-anchor queue whose first looks are consumed by the
walk (markAnchorRotationServed() after descent, requeueStaticAnchorFreshFront()
for unwalked picks, starvation drain only when the fresh budget is exhausted),
tiered selection with the cap-skipped leak-suffix spending, the root-enumeration
retry-stamp windows, and the gate-before-join Profiler::stop(). The merged
collectStaticFieldAnchorsForRotation() and heapReferenceCallback() are this
branch's coherent bodies plus the explicitly ported pieces above.

Main tests dropped (machinery deliberately not adopted):
- ReferenceChainsMergeAtRiskTest.MergedAnchorsAppearOnce: pins main's
  mergeAtRiskAnchors() helper; this branch merges the drained at-risk anchors
  inline in the pass, covered by RotationSelectionBoundaryIsPersistentPlusDrained.
- TransientNonContainerAnchorsArePruned: pins main's transient_root indexing of
  unknown locals; this branch parks unknown locals in _pending_local_anchors and
  prunes them via compactLocalAnchors()/resolvePendingLocalAnchors().
- FreshAnchorQueueDropsOldestPastCap: pins main's fresh-queue overflow drop
  counter (cap eviction + budget-exhausted drain clear both bump
  REFERENCE_CHAIN_STATIC_FRESH_DROPPED); this branch never pushes past
  STATIC_ANCHOR_FRESH_CAP and keeps queued turns when the fresh budget is
  positive, so the counter semantics do not apply.
- SelectLeakCandidatesTest.ScratchOverflowEvictsFirstMinimalSlot,
  ScratchOverflowEvictsTheGlobalMinimumThread, and
  LivenessTrackerScratchTest.{IncomingAndReappearingTidsStartFromTheirOwnObject,
  LowestCountIsEvictedEvenWithMoreAges, FewerAgesLoseAtEqualCount,
  CompleteTieEvictsFirstSlot}: pin main's immediate newcomer-takeover victim
  scan in the per-klass scratch; this branch's insertThreadGen() always stages
  newcomers in the challenger pool and promotion must outrank the weakest
  incumbent (2122bf2, "takeover earns against the victim's accumulated
  evidence").

Main tests adapted to this branch's semantics (not dropped):
- InWindowPruneKeepsFreshQueueSelections,
  FairCursorAfterPruneDelaysSkippedSurvivorsOneLap: selection skips dead
  entries but never prunes/compacts the index (dead entries leave at
  walk-resolution time), so the trailing index assertions and the
  cursor-lap expectations were re-derived.
- StaticFieldCollectionRemainsIndexedAfterFirstLook: the confirmed collection
  is re-enlisted through the journal on the sweep's next lap (one extra
  scripted edge + flush) instead of main's direct first-sight indexing.
- DeadAnchorIndexEntriesArePruned: pruning asserted at walk-resolution time
  (mock dead_tags -> removeMissingAnchors()) instead of at selection.
- StaticEdgeRefreshesUnknownClassOfIndexedHolder: kept as written; the
  0 -> resolved upgrade fires through the relaxed REENLIST record + replay.
- FirstPassWatchesQualifiedLeakBeforeCompleting: the idempotent second-poll
  assertions moved before stop() - this branch's stop() forces the terminal
  RECORDING_END state and clears candidates, so the post-stop expectations
  from main cannot hold.

Merge-splice repairs: restored promoteThreadGensForTest()'s closing brace in
LivenessTrackerTestAccessor (the merge nested main's Scratch/findThread members
inside its body) and removed three byte-identical duplicate
SelectLeakCandidatesTest definitions the merge had kept from both sides.
The merge left the anchor-index documentation split between main's
selection-order section and this branch's shares/eviction section. Fold the
unique facts of both into the maintained section: the UNKNOWN_TRUNCATED shape
contract, the tag-keyed class-tag upgrade view, and selection skipping (not
pruning) dead entries.
eraseAnchorEntries() position rebuild skips a tag missing from the class-tag
map instead of dereferencing end(); set and map are maintained together, so
the guard only turns a hypothetical divergence into a skip. clearPrefix()
clears the whole journal (its only caller replays the full snapshot), so the
comment says that instead of promising a prefix clear.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Two distinct logic gaps are flagged: in referenceChainTraversal.cpp, newly admitted JNI-global roots with an initial tag of 0 are never journaled (no DURABLE_ADD), so they miss targeted anchor rotation until a subsequent enumeration; in referenceChainWalk.cpp, static-field holders with unclassified class shapes are rejected by the selectability check and recorded only as AT_RISK, leaving them out of all reconciliation inputs and causing them to miss persistent container rotation after their FIFO walk.

Open Bits AI session

🤖 Bits Code Review · Commit 343d099

@jbachorik jbachorik added the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
@jbachorik

jbachorik commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you.

This run
  • 20:31 · started · CI on head 343d09913 in progress · nightly slow-matrix run 45065274198 (testSlowDebug, 118 jobs) being watched for PROF-16825
  • 20:52 · CI red · nightly 45065274198 failed: LeakTagCorrelationReferenceChainTest.shouldCorrelateLeakChainTargetTagWithLiveObjectLeakTag on musl-11-librca-debug-amd64-slow and glibc-25-debug-aarch64-slow (not one of the two PROF-16825 tests); one flaky rerun issued
  • 21:15 · answered · 4 Bits threads — 1 fixed by 9345d3c43, 3 deferred with follow-ups named
  • 21:58 · staged · 4 commits on the babysit branch for the session's verified push — d74eeca3b, 4d0048dad, 870e07450, 0c23ea5c3 (6 more Bits threads answered with the staged shas)
  • 22:12 · retried · nightly rerun of both failed cells completed green — the correlation failure classifies as flaky; both PROF-16825 tests passed in every completed cell
  • 22:12 · CI green · PR-CI run 45066903748 (regular suite) completed green on 343d09913

Stop summary (22:20 UTC): no commits pushed by this run — 4 SSH-signed commits staged on local branch babysit-pr-899 (base 343d09913): d74eeca3b (queued first looks through selection/saturation), 4d0048dad (fresh durable roots journaled on first admission), 870e07450 (stale join-count assertion dropped), 0c23ea5c3 (displaced incumbent's live sample kept); handover patch at build/handover-pr899-9f14ff62.mbox in the worker worktree, push + pre-pr-review owned by the session (no C++ toolchain / no headless reviewer in the worker sandbox). All 10 open Bits threads answered, one signature reply each; threads left for the human to resolve. Findings deferred with named follow-ups: 4642909857 (flat-cohort takeover semantics), 4645245981 (per-source retry stamps), 4659826806 (static-report shape seeding). bits_triggered/bits_rerequested empty — @datadog review re-request held: the session's push creates the new head that re-reviews. Old-pipeline reds (dd-gitlab/default-pipeline, post-pr-comment, DDCI, bits-review/head on 343d09913) are expected to clear via the pending push.

✅ stopped 22:20 UTC · nightly slow matrix green after flaky rerun (PROF-16825 tests passed in every cell) · PR CI green on 343d09913 · 10 Bits threads answered · 4 staged commits handed to the session for verified push

@jbachorik jbachorik removed the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
- fair-tier picks hand their queued first look to the caller's requeue
- saturated passes keep non-indexed/evicted fresh-queue entries pending
- promotion preserves displaced incumbents' epoch samples
- heapRootCallback journals DURABLE_ADD on fresh durable admission
- drop stale joinCallCountForTest assertion after the custom hook

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

PR #899 targets reference-chain nightly flakes and caps the anchor index; this revision restores unwalked fair-tier turns, displaced incumbent samples, and first-report JNI-global indexing. Three P2s remain: static attachments can fail to trigger class-shape classification (deferred), repeat JNI-global reports can skip class-tag refresh, and live threads that stay in the challenger pool can lose leak qualification.

Open Bits AI session

🤖 Bits Code Review · Commit e0fd8a5

if (durable_root && *tag_ptr > 0 && !isLeakTag(*tag_ptr)) {
completed_durable_report = ctx->frontier->rootReportCompleted(
*tag_ptr, translated_root_kind,
/*anchor_indexed=*/ctx->tracker->isAnchorIndexed(*tag_ptr),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Refresh missing class tags on repeat JNI-global reports

If a JNI-global collection is indexed with class tag 0, a later JNI-global report carrying its resolved class tag still counts as complete because the holder is indexed. The callback skips the journal record that would refresh the index, whereas the base refreshed it on repeat reports. If no other anchor supplies that class tag, reconciliation never classifies the holder, and reparenting removes its root-attached fallback for rotation. Allow a resolved class tag to refresh an indexed zero-tag entry through the journal while retaining retry suppression for reports with no new information.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

scratch.threads[victim] = candidate;
}
}
scratch.challenger_count = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Fold live trends for challengers that do not promote

If 16 threads with nine or more retained samples fill the incumbent array before an already-qualified thread contributes eight samples, the latter stays in the challenger pool. Resetting the pool discards its epoch sample, so the trend fold records zero and clears its qualification even though eight samples meet the retained-count bar. The base would admit this late thread by replacing a scratch entry and record its eight samples. Carry unpromoted samples into the existing-trend fold as well as displaced samples, without creating new trend entries for the unpromoted threads or recording an epoch twice.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI human_review:recommended Human review recommended — see PR description

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants