Skip to content

Automated release train: nightly certification, weekly minor release, dd-trace-java bump (PROF-16810) - #1037

Open
jbachorik wants to merge 32 commits into
mainfrom
drydock/jb-keen-pearly-heron
Open

jbachorik wants to merge 32 commits into
mainfrom
drydock/jb-keen-pearly-heron

Conversation

@jbachorik

@jbachorik jbachorik commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Adds a fully automated release train on top of the existing tag-based release flow:

  • Nightly certification (.github/workflows/certification.yml, 04:00 UTC) runs the release-validated.yml pre-release gate (testDebug+testAsan, regular and slow, gtests), the fuzz job, and reads the dd-trace-integration cells from the commit's GitLab pipeline. A green run attaches a git note on the certified commit under refs/notes/certification; a failed run notifies Slack and files a high-priority PROF ticket (deduped per commit).
  • Friday release train (.github/workflows/release-train.yml, Fri 11:00 UTC) picks the newest certified commit the last v_* tag does not cover, and dispatches release-validated.yml pinned to that commit (minor, dry_run=false). It never releases twice from the same certified commit and never releases a commit older than the last release; both rules are enforced in pick-release-candidate.sh and re-checked before dispatch. Skips with a Slack note otherwise.
  • dd-trace-java bump (update_assets.yml + bump-ddtrace.sh): once the release artifacts are live on Maven Central and the GitHub release is published, a draft PR on DataDog/dd-trace-java bumps gradle/libs.versions.toml and dd-java-agent/ddprof-lib/gradle.lockfile (the exact two files a manual bump touches) with labels and a link to the release notes. A human merges it.

Both new workflows accept manual workflow_dispatch runs; the train takes optional sha and dry_run inputs for controlled trials. Major and patch releases stay manual. release-validated.yml and release.sh now accept a commit on main as the release ref (pinned dispatch); the manual flow is unchanged.

Docs: doc/building/ReleaseTrain.md (+ index, Versioning.md, utils/README.md, AGENTS.md pointer).

Motivation:

Releases and dd-trace-java bumps are operator-driven today; the release gate re-runs the same suite a nightly already ran. The train makes the weekly minor release hands-off while keeping the published-artifact guarantee: only commits that passed the full certification suite can be released. PROF-16810 records the design decisions.

Additional Notes:

Review: agentic Sphinx review (16 range reviews across the fix rounds) plus eight Bits trusted-bot review rounds and one human review round (12 + 6 + 4 + 4 + 5 + 2 + 4 + 12 findings, all addressed). The initial Sphinx review found 19 findings (3 HIGH: a failed dispatched release did not fail the train, and scheduled runs dispatched dry_run= with an empty boolean); the Bits round caught a further 12 (missing base_branch/target_sha job outputs, source_sha reachable as shell input, dispatch-ref pinning semantics, retired Jira search endpoint, mainline-vs-maintenance tag gating); every fix was re-reviewed and its acceptance brief verified; the final fix rounds found 0 HIGH/MEDIUM findings.

Bits rounds 3–5 caught: artifact-name collisions when one run invokes the test workflow repeatedly (now disambiguated by an artifact_suffix threaded through to the cache and failure artifacts), a GitLab page-fetch failure reading as a complete listing, a queued duplicate dispatch double-releasing a certified commit, the pinned source SHA reaching the checkout command uncanonicalized (tag names and revision expressions), manual mainline releases racing the train's concurrency group, retag recovery blocked by the new duplicate-release guard, an unserialized dependency-bump race, notification steps skipping when only a dependency job failed, and pipeline selection returning a trigger pipeline without the integration jobs. A follow-up Sphinx review of those fixes verified all six sub-claims and caught two regressions (page order under sort=desc hides the push pipeline; step-level failure() skips notifications), both fixed. One open LOW: pick-release-candidate.sh leaks its mktemp files if it aborts between creation and cleanup (pre-existing pattern, ephemeral CI runner). Two late Sphinx fix-review sessions wedged before delivering their sealed reports (invalid_state in the verification phase); their completed rounds found only LOW/MEDIUM items, and one undelivered MEDIUM from the final certification-workflow round remains unexamined — flagged for the next reviewer.

Review trail: initial 1d5a5519..bfab1a00d, then per-fix ranges through 1a38bf2a..d653ef961 (the review marker lists the ranges). Human review is recommended before merging: this change automates the release path and adds consumers of four new repo secrets (GITLAB_API_TOKEN, ATLASSIAN_BASE_URL/ATLASSIAN_USER_EMAIL/ATLASSIAN_API_TOKEN, DDTRACE_GH_TOKEN) that must be provisioned before the automation is live; certification cannot certify without the GitLab token by design.

How to test the change?:

  • .github/scripts/tests/test_release_train.sh — hermetic (23 assertions): note creation, idempotency, and every candidate-selection rule including the no-double-release and no-older-than-last-release guards, multi-note tip selection.
  • .github/scripts/tests/test_release_automation.sh still passes (44 groups) against the modified release.sh; both run in ci.yml with bash -n and shellcheck.
  • qualityChecks passes, including the doc index.
  • End-to-end after merge: gh workflow run certification.yml on main certifies HEAD; gh workflow run release-train.yml -f dry_run=true picks the certified commit and dispatches a dry-run release (no tag, no GitLab trigger).

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16810

certify.sh attaches a certification note on refs/notes/certification and
pushes it; pick-release-candidate.sh selects the newest certified commit
the release train has not released yet. test_release_train.sh covers note
creation, idempotency, and every selection rule.
Certifies main HEAD with the release-validated gate (testDebug/testAsan,
regular and slow, gtests), fuzz, and the dd-trace-integration cells read
from the commit's GitLab pipeline. Green runs attach a certification note;
failures notify Slack and file a PROF ticket.
The release train picks the newest unreleased certified commit and
dispatches release-validated.yml pinned to it. release-validated.yml and
release.sh accept a commit on main as the release ref; the manual flow is
unchanged. ci.yml runs the new hermetic tests.
update_assets.yml runs bump-ddtrace.sh once the release artifacts are live
on Maven Central; the script bumps the ddprof version in libs.versions.toml
and the lockfile and opens a draft PR.
Fail the train when the dispatched release fails (an ::error:: annotation
does not fail the step) and default dry_run on scheduled runs, which left
the dispatch with an empty boolean input. Pick the newest certified commit
by history position instead of committer date, fail loudly on notes-fetch
errors, allow non-fast-forward notes updates, and let the bump job treat
the missing-token exit as a documented skip.
Select the tip-most certified commit with awk over rev-list --topo-order:
the grep|head pipeline aborted under set -euo pipefail once more than one
certified commit existed (grep exit 1 on no match, SIGPIPE on many).
Distinguish a missing notes ref (ls-remote exit 2) from a transport
failure; URL-encode the GitLab ref filter with a sha-only fallback;
disambiguate the dispatched-run lookup by dispatch time and raise the
retry budget; report the train summary on failure; push the bump branch
before recreating it.
Absorb runner/GitHub clock skew in the dispatched-run filter with a
5-minute margin; fail loudly when rev-list fails instead of reporting no
candidate; guard the bump re-push after a branch recreate.
The title timestamp window could misattribute a concurrent manual
dispatch of the same candidate; request_id is unique to this train's
dispatch and appears in the run title.
The run title ends with a closing bracket after the request_id, so
endswith never matched.
last | .databaseId on an empty array prints the literal null, which
broke the retry loop on its first attempt.
Prevents a longer concurrent run id with the current one as a prefix
from matching.
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@jbachorik jbachorik added human_review:recommended Human review recommended — see PR description dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates labels Nov 20, 2026
CI's shellcheck fails SC2015 on it.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Certification is rejected by workflow validation, and a missing job-output mapping breaks automated and manual releases. Additional defects affect GitLab matrix evaluation, release-candidate safeguards, dependency updates, and failure notifications.

Open Bits AI session

🤖 Bits Code Review · Commit b618429

Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/scripts/certify.sh Resolved
Comment thread .github/workflows/release-train.yml Resolved
Comment thread .github/scripts/file-jira-issue.sh Resolved
Comment thread .github/scripts/file-jira-issue.sh Resolved
Comment thread .github/workflows/release-train.yml Resolved
Comment thread .github/scripts/bump-ddtrace.sh Resolved
Map base_branch and target_sha into the validate-inputs job outputs; pin
the release commit through source_sha with ref=main (the dispatch API
takes a branch ref, not a SHA); run version computation at the release
commit; move secrets checks into scripts; aggregate reusable-workflow job
conclusions; configure the git identity for the certification note; keep
script diagnostics on stderr; gate candidate selection on mainline tags
only and apply the same rule to manual overrides; use the /search/jql
endpoint with an ADF comment body; refuse a ddprof pin downgrade; compose
the Slack message from explicit outputs.
@dd-octo-sts

dd-octo-sts Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #45073230513 | Commit: 5d3b99f | Duration: 21m 40s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-22 22:04:43 UTC

Pass the release commit to test_workflow.yml through a new ref input so
the pre-release gate tests the certified commit, not main HEAD. Pass
source_sha through env (dispatcher-controlled input must not be
interpolated into the shell program). Require override candidates to be
reachable from origin/main.
Remove the duplicated ref key in the musl-aarch64 checkout; record and
pass the tested commit (inputs.ref) in test metadata and the aarch64
docker script instead of the dispatch ref.
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Certification cannot recognize the existing GitLab matrix jobs, preventing new certified release candidates. The release and dependency-bump paths also contain maintenance-commit validation and Git authentication defects.

Open Bits AI session

🤖 Bits Code Review · Commit 826797b

Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/scripts/bump-ddtrace.sh Resolved
Comment thread .github/workflows/release-train.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/scripts/bump-ddtrace.sh Resolved
Match GitLab matrix-expanded integration cells by prefix and aggregate
cell statuses; paginate the job lookup; validate pinned patch/retag
commits against the release branch; include the run attempt in the
dispatch request_id; wire the token into git credentials for the bump
push; abort the bump on a PR-lookup failure.
gh api rejects --slurp with --jq: paginate per page and aggregate
conclusions in awk. Follow GitLab job pagination for matrix-expanded
pipelines; classify terminal non-success cells (failed, canceled,
skipped, manual) as failed instead of spinning on them; match matrix
cells by name or its ': [' suffix. Bump PR lookup falls back to empty
instead of the literal null. Distinguish merge-base errors from
not-an-ancestor. Serialize release runs per candidate with a
release-validated concurrency group.
@dd-octo-sts

dd-octo-sts Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

❌ 38 passed, 0 failed, 2 without a result out of 40 expected configurations

Test Matrix

Platform JDK 8 JDK 11 JDK 17 JDK 21 JDK 25
glibc-x64-hotspot ❔ ✅ ✅ ✅ ✅
glibc-x64-openj9 ❔ ✅ ✅ ✅ ✅
glibc-arm64-hotspot ✅ ✅ ✅ ✅ ✅
glibc-arm64-openj9 ✅ ✅ ✅ ✅ ✅
musl-x64-hotspot ✅ ✅ ✅ ✅ ✅
musl-x64-openj9 ✅ ✅ ✅ ✅ ✅
musl-arm64-hotspot ✅ ✅ ✅ ✅ ✅
musl-arm64-openj9 ✅ ✅ ✅ ✅ ✅

No result produced

These configurations wrote no validation log, so their outcome is unknown and
they are counted as failures. The job usually timed out or failed during setup;
see its log in the pipeline.

  • glibc-x64-hotspot-jdk8
  • glibc-x64-openj9-jdk8

Links

String-concatenating page arrays produced invalid JSON, and hoisting the
fetch out of the retry loop froze cell statuses on the first attempt.
Suffix cache_java artifact names per caller so the four certification
suite invocations in one run do not collide on sdkman-installation-*;
fail the GitLab job fetch on curl errors instead of masquerading as an
empty complete listing; file the PROF ticket even when Slack fails;
refuse a release commit that already carries a tag (queued duplicate
dispatch after a concurrent release).
The two pre-release test calls share one run and need distinct
artifact_suffix values like certification. Notify Slack and File PROF
ticket run on failure only, not on cancellation. Pre-seed the GitLab
STATUS map so an all-fetch-failures path reports unknown instead of an
unbound-variable abort. Avoid the dangling hyphen with an empty
artifact_suffix.
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

An accepted source revision can execute shell substitution after the release SSH key is loaded, even during a dry run. Release serialization, retag recovery, and concurrent dependency bumps also need fixes.

Open Bits AI session

🤖 Bits Code Review · Commit d7f22c8

Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/update_assets.yml Resolved
@jbachorik
jbachorik marked this pull request as ready for review November 20, 2026 17:25
@jbachorik
jbachorik requested a review from a team as a code owner November 20, 2026 17:25
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T17:30:19.396230Z d7f22c8 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7f22c8ec6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/release-validated.yml Resolved

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

A failed certification can finish its reporting job without sending Slack or filing a ticket. The release workflow also blocks retag recovery when publishing failed at the unchanged release commit.

Open Bits AI session

🤖 Bits Code Review · Commit d7f22c8

Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/update_assets.yml Resolved
@jbachorik jbachorik removed the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 20, 2026
Mainline releases share one concurrency group: a queued run validates
before the previous run's tag exists, so per-SHA grouping lets two
dispatches release the same commit. The pinned source is canonicalized
to its commit ID so a tag name or revision expression can neither
defeat the duplicate-release guard nor reach the release job's
checkout command. retag is exempt from the duplicate-release guard:
it re-points an existing tag at the same commit by design. The bump
job serializes against overlapping manual runs. Notification steps run
on !cancelled() (the job is gated on failure() || cancelled();
step-level failure() is false when only a dependency failed). Pipeline
selection prefers the push pipeline over same-SHA trigger pipelines
that lack the integration jobs.
GitLab pipeline selection sorts ascending: the push pipeline is the
oldest for a SHA and sort=desc would push it off the first page behind
the trigger pipelines it spawned. The unresolvable-source error keeps
the dispatcher's original input. The update-assets workflow serializes
per tag so a tag-push run and a manual run for the same version cannot
race the asset uploads and the --latest flag.
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR automates nightly certification, weekly minor releases, and dd-trace-java dependency bumps.
All previously reported issues are fixed: source-ref normalization,
retag recovery, certification notifications, GitLab pipeline selection,
and concurrent dependency-bump branch deletion;
no findings remain open.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit e5e0337

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The train can publish an older certified commit after a newer manual major release completes while it waits for the shared release lock. The duplicate-tag guard does not enforce release ordering.

Open Bits AI session

🤖 Bits Code Review · Commit e5e0337

Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
…tract

A queued mainline dispatch that predates a newer release (tagged while
it waited for the concurrency lock) passed the equality-only duplicate
guard; the mainline guard now also rejects a release commit that does
not contain a v_* tag already on main. Certification and release-train
Slack payloads carry the scenarios and failed_run_url fields that the
nightly.yml receiver consumes, alongside text.
One pass over the release tags now runs both guards: the duplicate-release
equality check and the staleness check against the release branch's tags
(main for major/minor, the release branch for patch), so a queued dispatch
cannot publish a commit older than the newest tag on its target branch.
The release train's skip notification stays plain text: the nightly
receiver treats the scenarios/failed_run_url fields as a failure report.
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR automates certification, weekly releases, and dd-trace-java bumps;
prior fixes cover source pinning, retag recovery, notifications,
pipeline selection, and bump concurrency.
The new release-position guard rejects major/minor dispatches
from tags by resolving nonexistent remote branches.

Open Bits AI session

🤖 Bits Code Review · Commit 689a7e7

Comment thread .github/workflows/release-validated.yml Resolved

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

A stale-branch manual dispatch can select an older commit for release after a newer mainline release. A local reproduction confirmed that validation accepts this case; the suggested replacement rejects it while preserving maintenance-branch handling.

Open Bits AI session

🤖 Bits Code Review · Commit 689a7e7

Comment thread .github/workflows/release-validated.yml Resolved

@zhengyu123 zhengyu123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review findings on 689a7e7. Each was checked against the code at this commit. Items 1–3 can break a release or certification; the rest are smaller.

  1. Retag can rewind a release branch (release-validated.yml).
  2. Older-line patch releases end with a red bump-dd-trace-java job (update_assets.yml).
  3. fetch_all_jobs loses jobs past 100 (certification.yml).
  4. bump-ddtrace.sh fails when the lockfile is absent.
  5. Smaller items: RELEASE_BASE_BRANCH not exported, runbook fallback, job timeout, test coverage, duplicated comment.

Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/workflows/update_assets.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/workflows/certification.yml Resolved
Comment thread .github/scripts/bump-ddtrace.sh Resolved
Comment thread .github/workflows/release-validated.yml Resolved
Comment thread .github/scripts/release.sh Resolved
Comment thread doc/building/ReleaseTrain.md Resolved
Comment thread .github/workflows/release-validated.yml Resolved
@jbachorik jbachorik added the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
@jbachorik

jbachorik commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you.

This run
  • 20:00 · started · CI on head 689a7e738 — test matrix green, bits-review/head FAIL (1 P1 finding) and codeowner-review/head failing (sensitive .github/ paths, human approval pending)
  • 20:20 · answered · 12 new trusted threads (Bits P1+P2 on release-validated.yml:156, zhengyu123 ×10 across release CI) — all confirmed real, all in human-gated .github/ scope, queued for manual application
  • 20:20 · Bits · FAIL verdict stands on 689a7e738 — clearing it needs the human-applied CI fixes above, not a re-trigger
  • 20:30 · CI green · test-matrix rerun 45063983077 completed successfully on 689a7e738
  • 20:40 · answered · Bits re-review posted 4 new threads restating already-answered findings — each replied once with a pointer to the original thread
  • 20:40 · Bits · FAIL stands — no new code to review, findings await human-applied CI fixes
  • 20:51 · stopped · blocked on human action — 16 findings answered, 0 fix-push rounds used

✅ stopped 20:51 UTC · blocked on human action: 16 confirmed findings in human-gated .github/ CI scope await manual fixes; test matrix green on 689a7e7; bits-review FAIL + codeowner approval pending

Stop summary

Commits pushed: none — every finding landed in CI configuration/scripts or docs outside the babysitter's edit scope (sensitive .github/ paths also gated on pending @DataDog/profiling-java approval), so no fix-push cycle was run.

Findings flagged for manual application (all confirmed against the code):

  • release-validated.yml:156 — tag pass must compare against the effective target branch (origin/main for major/minor); Bits P1+P2, duplicated by re-review
  • release-validated.yml:293 — retag can rewind the release branch (fresh-clone force-with-lease)
  • update_assets.yml:109 / bump-ddtrace.sh:63 — downgrade refusal exits 1 after assets are published
  • certification.yml:174 — jq page merge via argv can exceed 128 KiB and fail silently (reproduction-confirmed)
  • certification.yml:101 — 240-min job timeout below the 360-min retry budgets
  • bump-ddtrace.sh:93 — git add on an optional lockfile exits 128
  • release-validated.yml:309 / release.sh:26 — RELEASE_BASE_BRANCH never reaches release.sh
  • doc/building/ReleaseTrain.md:93 — documented recovery path is rejected by release-train gates
  • test_release_train.sh:8 — coverage gap for bump-ddtrace/file-jira-issue/release.sh/pick-release-candidate
  • release-validated.yml:136 — duplicate comment paragraph (nit)

Threads left for the author: none unanswered — all 16 new trusted threads this run carry exactly one babysit reply; open threads from earlier rounds were already answered before this run.

bits_triggered heads: d653ef9, e5e0337, 689a7e7 (no new head this run; the FAIL re-review of 689a7e7 was an external re-trigger).

@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Pinned retagging can rewind a maintenance branch. The release-age guard also checks the dispatch branch rather than main, allowing a mainline release to publish older code.

Open Bits AI session

🤖 Bits Code Review · Commit 689a7e7

git config --global user.email "java-profiler@datadoghq.com"
git config --global user.name "Datadog Java Profiler"
git checkout -B "$GITHUB_REF_NAME" "$GITHUB_SHA"
git checkout -B "${{ needs.validate-inputs.outputs.base_branch }}" "${{ needs.validate-inputs.outputs.target_sha }}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Reject retag targets behind the release branch tip

For an unpublished release, retag validation accepts a source_sha behind the maintenance branch tip. This checkout resets the local branch to that historical commit, and release.sh then force-pushes it. Because the fresh clone's tracking ref matches the current remote tip, --force-with-lease permits the rewind, removing subsequent commits from the remote branch. Check the target against the freshly cloned branch tip before resetting it.

Suggested change
git checkout -B "${{ needs.validate-inputs.outputs.base_branch }}" "${{ needs.validate-inputs.outputs.target_sha }}"
if [ "${{ inputs.release_type }}" == "retag" ] && [ "${{ needs.validate-inputs.outputs.target_sha }}" != "$(git rev-parse "origin/${{ needs.validate-inputs.outputs.base_branch }}")" ]; then
echo "::error::Retag must target the current tip of the release branch"
exit 1
fi
git checkout -B "${{ needs.validate-inputs.outputs.base_branch }}" "${{ needs.validate-inputs.outputs.target_sha }}"

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the thread on this line from the earlier review pass: retag validation accepts a source_sha behind the release branch tip, the checkout resets to it, and release.sh's fresh-clone --force-with-lease permits the rewind. Checking the target against the freshly cloned branch tip is queued for manual application (CI config, human-gated).

Comment thread .github/workflows/release-validated.yml Outdated
exit 1
fi
RC=0
git merge-base --is-ancestor "$TAG_COMMIT" "origin/$BRANCH" || RC=$?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Check release age against the effective target branch

Major/minor dispatches now accept a source_sha on main regardless of the dispatch branch, but this filter still checks tags against that dispatch branch. Newer mainline tags absent from a stale branch are skipped, allowing the age guard to approve older source—for example, a major release computed as 2.0.0 from before the latest 1.33.0 release. Compare against origin/main for major/minor releases.

Suggested change
git merge-base --is-ancestor "$TAG_COMMIT" "origin/$BRANCH" || RC=$?
RELEASE_TARGET_BRANCH="main"
if [ "$TYPE" == "patch" ] || [ "$TYPE" == "retag" ]; then
RELEASE_TARGET_BRANCH="$BRANCH"
fi
git merge-base --is-ancestor "$TAG_COMMIT" "origin/$RELEASE_TARGET_BRANCH" || RC=$?

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the two threads on this line from the earlier Bits pass: the tag-ancestry filter must run against the effective target branch (origin/main for major/minor, origin/$BRANCH for patch/retag), else newer mainline tags are skipped on stale dispatch branches. Queued for manual application (human-gated .github/ scope).

Comment thread .github/workflows/certification.yml Outdated
if [ -z "$jobs_json" ]; then
jobs_json="$page_json"
else
jobs_json=$(jq -cn --argjson a "$jobs_json" --argjson b "$page_json" '$a + $b')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Merge GitLab job pages through stdin

Passing accumulated job JSON through --argjson exceeds Linux's per-argument limit once a page grows beyond approximately 128 KiB. A focused reproduction using 100 jobs with 1,500-byte commit messages produced 'Argument list too long', yet the caller accepted an empty listing because the conditional function call suppresses errexit. Required integration jobs then appear missing and certification fails. Merge through stdin and propagate jq failures.

Suggested change
jobs_json=$(jq -cn --argjson a "$jobs_json" --argjson b "$page_json" '$a + $b')
jobs_json=$(printf '%s\n%s\n' "$jobs_json" "$page_json" | jq -sc 'add') || return 1

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the thread on this line from the earlier pass — now with a focused reproduction confirming 'Argument list too long' and the silently accepted empty listing. Merging pages through stdin and propagating the jq failure matches what is queued for manual application.

Comment thread .github/scripts/bump-ddtrace.sh Outdated
Comment on lines +62 to +63
echo "ERROR: refusing to downgrade ddprof from ${OLD_VERSION} to ${VERSION}" >&2
exit 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Skip dependency bumps for older maintenance releases

The asset workflow invokes this script for maintenance releases too. If dd-trace-java already pins 1.33.0, publishing 1.32.2 reaches this exit 1, making the workflow fail after successfully publishing the release and assets. Preserving the newer dependency is expected behavior and should return a successful skip, as the already-pinned case does.

Suggested change
echo "ERROR: refusing to downgrade ddprof from ${OLD_VERSION} to ${VERSION}" >&2
exit 1
echo "dd-trace-java already pins newer ddprof ${OLD_VERSION}; skipping ${VERSION}"
exit 0

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the thread on update_assets.yml:109 from the earlier pass: the downgrade refusal's exit 1 turns an already-published maintenance release red. Returning a successful skip, like the already-pinned case, is queued for manual application.

@jbachorik jbachorik removed the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
The retag path forced the release branch to the dispatch target, so a
retag of a non-tip commit rewound the branch and dropped commits;
retag now requires the current branch tip. The staleness guard
compared tags against the dispatch ref, which breaks for tag-named
refs and misses newer mainline tags on a stale branch; it now checks
main for major/minor and the dispatch branch for patch/retag. The
GitLab job-page merge moved to stdin (--argjson can exceed the
per-argument limit and fail silently) and the job timeout now exceeds
the combined retry budgets. bump-ddtrace.sh skips (exit 0) when
dd-trace-java already pins a newer version, adds the lockfile
conditionally, and uses portable sed -i; its flows are covered by new
hermetic tests. RELEASE_BASE_BRANCH reaches release.sh through the
step env, the doc's older-candidate fallback names a working path, and
the duplicated guard comment is gone.
The downgrade skip carries a ::warning:: so it stays visible in the
workflow run; the sed backups are removed on the verification-failure
paths too; the staleness-guard comment states that only major/minor
dispatches can carry a tag-named ref; the missing-lockfile test bumps
a version unique to its case so it cannot pass on the earlier case's
branch.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR automates nightly certification, weekly minor releases,
and dd-trace-java bumps, with prior fixes for source pinning,
notifications, pipeline selection, retag recovery, and bump concurrency.
The release-history guard now uses the effective target branch,
resolving tag-based dispatch failures; no review findings remain open.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 7128c34

@datadog-datadog-prod-us1-2

Copy link
Copy Markdown

Pipelines

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

Bits review gate | publish (7128c34)

View more details · View in GitHub Actions

DataDog/java-profiler | integration-test-x64-glibc: [hotspot, 8]

View more details · View in GitLab

DataDog/java-profiler | integration-test-x64-glibc: [openj9, 8]

View more details · View in GitLab

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7128c34 | Docs | View more details | Give us feedback!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

human_review:recommended Human review recommended — see PR description

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants