Repository navigation
Automated release train: nightly certification, weekly minor release, dd-trace-java bump (PROF-16810) #1037
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jbachorik
wants to merge
32
commits into
main
Choose a base branch
from
drydock/jb-keen-pearly-heron
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,699
−35
Open
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
71b4cf9
Add certification-note scripts and hermetic tests
jbachorik b2e04e1
Add nightly certification workflow
jbachorik aa71f71
Add weekly release train and pinned-commit release dispatch
jbachorik 6b8a629
Open a dd-trace-java bump PR after release publication
jbachorik bfab1a0
Document the automated release train
jbachorik 491085e
Fix release-train review findings
jbachorik d145d82
Harden release-train fix details
jbachorik 1d6d10f
Fix undefined TEMP_DIR in bump branch recreate
jbachorik 8670f5c
Guard transient failures in train scripts and run discovery
jbachorik c41716c
Correlate the dispatched release run by request_id
jbachorik e9286ae
Match the dispatched run title with contains
jbachorik e438b96
Return empty from jq when no dispatched run matches
jbachorik b0c7472
Match the request_id including its closing bracket
jbachorik b618429
Fix jq string quoting in the request_id match
jbachorik d9427d8
Avoid the A && B || C idiom in the train test
jbachorik 8a08f60
Address Bits review findings on the release train
jbachorik 9d9287a
Test the pinned candidate in pre-release jobs
jbachorik f2baf96
Pass source_sha through the step env
jbachorik 826797b
Fix test_workflow ref plumbing details
jbachorik 29b401f
Address second Bits review round
jbachorik 1a38bf2
Fix second-round review findings
jbachorik 5bc07a8
Fetch GitLab jobs inside the retry loop and merge pages with jq
jbachorik d653ef9
Classify GitLab cell statuses by terminal set and treat null conclusi…
jbachorik 4732999
Merge branch 'main' into drydock/jb-keen-pearly-heron
jbachorik ccc608e
Address third Bits review round
jbachorik d7f22c8
Suffix release-validated test calls and scope failure reporting
jbachorik d29c73c
Serialize mainline releases and canonicalize the pinned SHA
jbachorik e5e0337
Fix pipeline page order and notification error path
jbachorik b70442e
Reject stale queued mainline releases and honor the Slack payload con…
jbachorik 689a7e7
Merge the release guards into one tag pass and scope the skip payload
jbachorik 16d89e6
Fix release train findings from human review and Bits round 8
jbachorik 7128c34
Scope the downgrade skip and tighten the round-8 fixes
jbachorik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # Copyright 2026, Datadog, Inc | ||
|
|
||
| # Bump the ddprof dependency in DataDog/dd-trace-java and open a draft PR. | ||
| # | ||
| # Environment: | ||
| # DDTRACE_GH_TOKEN token with contents:write + pull_requests:write on | ||
| # DataDog/dd-trace-java (required) | ||
| # | ||
| # Usage: | ||
| # bump-ddtrace.sh <new-version> [release-notes-url] | ||
| # | ||
| # The change is exactly two files: gradle/libs.versions.toml (the ddprof | ||
| # version.ref) and dd-java-agent/ddprof-lib/gradle.lockfile. Skips (exit 0) | ||
| # when an open PR from a previous run already bumps to this version. | ||
|
|
||
| if [ $# -lt 1 ]; then | ||
| echo "Usage: $0 <new-version> [release-notes-url]" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| VERSION="$1" | ||
| NOTES_URL="${2:-https://github.com/DataDog/java-profiler/releases/tag/v_${VERSION}}" | ||
| REPO="DataDog/dd-trace-java" | ||
| BRANCH="ddprof/bump-${VERSION}" | ||
|
|
||
| if [ -z "${DDTRACE_GH_TOKEN:-}" ]; then | ||
| echo "WARNING: DDTRACE_GH_TOKEN is not set; skipping dd-trace-java bump" >&2 | ||
| exit 3 | ||
| fi | ||
|
|
||
| export GH_TOKEN="$DDTRACE_GH_TOKEN" | ||
| export GH_REPO="$REPO" | ||
| # The token authenticates gh API calls but not the plain git push from the | ||
| # temp clone; wire it in as a git credential helper. | ||
| gh auth setup-git --hostname github.com >/dev/null | ||
|
|
||
| WORK_DIR=$(mktemp -d) | ||
| trap 'rm -rf "$WORK_DIR"' EXIT | ||
|
|
||
| gh repo clone "$REPO" "$WORK_DIR/repo" -- --depth 1 --no-tags | ||
| cd "$WORK_DIR/repo" | ||
|
|
||
| TOML="gradle/libs.versions.toml" | ||
| LOCK="dd-java-agent/ddprof-lib/gradle.lockfile" | ||
|
|
||
| OLD_VERSION=$(awk -F '"' '/^ddprof = / {print $2; exit}' "$TOML") | ||
| if [ -z "$OLD_VERSION" ]; then | ||
| echo "ERROR: no ddprof version entry found in $TOML" >&2 | ||
| exit 1 | ||
| fi | ||
| if [ "$OLD_VERSION" == "$VERSION" ]; then | ||
| echo "dd-trace-java already pins ddprof ${VERSION}; nothing to do" | ||
| exit 0 | ||
| fi | ||
|
jbachorik marked this conversation as resolved.
Resolved
|
||
| # A downgrade means a maintenance-line release while dd-trace-java already | ||
| # pins a newer mainline version; downgrading the pin is impossible, and the | ||
| # release itself published fine, so this is a skip, not a failure. | ||
| if [ "$(printf '%s\n%s\n' "$OLD_VERSION" "$VERSION" | sort -t. -k1,1n -k2,2n -k3,3n | head -1)" == "$VERSION" ]; then | ||
| # ::warning:: keeps the skip visible in the workflow run; the exit code | ||
| # alone is indistinguishable from a completed bump. | ||
| echo "::warning::dd-trace-java already pins newer ddprof ${OLD_VERSION}; skipping ${VERSION}" | ||
| exit 0 | ||
| fi | ||
|
|
||
| git checkout -B "$BRANCH" | ||
|
|
||
| # The .bak suffix form is the only sed -i variant that is identical on GNU | ||
| # and BSD; the backups are removed right after the verification greps. | ||
| sed -i.bak "s/^ddprof = \"${OLD_VERSION}\"/ddprof = \"${VERSION}\"/" "$TOML" | ||
| if ! grep -q "^ddprof = \"${VERSION}\"" "$TOML"; then | ||
| rm -f "$TOML.bak" | ||
| echo "ERROR: failed to update $TOML" >&2 | ||
| exit 1 | ||
| fi | ||
| rm -f "$TOML.bak" | ||
|
|
||
| if [ -f "$LOCK" ]; then | ||
| sed -i.bak "s/^com\.datadoghq:ddprof:${OLD_VERSION}=/com.datadoghq:ddprof:${VERSION}=/" "$LOCK" | ||
| if ! grep -q "^com\.datadoghq:ddprof:${VERSION}=" "$LOCK"; then | ||
| rm -f "$LOCK.bak" | ||
| echo "ERROR: failed to update $LOCK" >&2 | ||
| exit 1 | ||
| fi | ||
| rm -f "$LOCK.bak" | ||
| fi | ||
|
|
||
| # An open PR from a previous run means the bump is already tracked; a lookup | ||
| # failure must abort — treating it as "no PR" could delete an existing PR's | ||
| # head branch in the recreate fallback below. | ||
| EXISTING=$(gh pr list --head "$BRANCH" --state open --json number --jq '.[0].number // empty') | ||
| if [ -n "$EXISTING" ]; then | ||
| echo "Open bump PR already exists: #$EXISTING" | ||
| exit 0 | ||
| fi | ||
|
|
||
| git config user.email "java-profiler@datadoghq.com" | ||
| git config user.name "Datadog Java Profiler" | ||
| git add "$TOML" | ||
| [ -f "$LOCK" ] && git add "$LOCK" | ||
| git commit -m "Bump java-profiler (ddprof) to ${VERSION}" | ||
| # The branch is bot-owned and version-scoped. Push plainly first; only a | ||
| # non-fast-forward reject (a stale branch from a closed previous run) | ||
| # recreates it, so a concurrent PR on the branch is not deleted by accident | ||
| # and a delete failure is not hidden. | ||
| if ! git push origin "$BRANCH" 2>"$WORK_DIR/push.err"; then | ||
| if grep -qE 'non-fast-forward|fetch first' "$WORK_DIR/push.err"; then | ||
| git push origin :"$BRANCH" || { echo "ERROR: failed to recreate $BRANCH" >&2; exit 1; } | ||
| git push origin "$BRANCH" || { echo "ERROR: failed to push recreated $BRANCH" >&2; exit 1; } | ||
| else | ||
| cat "$WORK_DIR/push.err" >&2 | ||
| exit 1 | ||
| fi | ||
| fi | ||
|
|
||
| PR_BODY=$(cat <<EOF | ||
| # What Does This Do | ||
|
|
||
| Bumps java-profiler (ddprof) from ${OLD_VERSION} to ${VERSION} and updates the dependency lockfile. | ||
|
|
||
| Automated bump by the java-profiler release train, opened after the release | ||
| artifacts were published. See the | ||
| [release notes](${NOTES_URL}) for the changes introduced since ${OLD_VERSION}. | ||
| Refine this description before merging if a curated summary of notable | ||
| changes is needed. | ||
| EOF | ||
| ) | ||
|
|
||
| PR_URL=$(gh pr create --draft --title "Bump java-profiler (ddprof) to ${VERSION}" \ | ||
| --body "$PR_BODY" --label "type: bug fix" --label "comp: profiling" --label "tag: ai generated") | ||
|
|
||
| echo "Opened $PR_URL" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # Copyright 2026, Datadog, Inc | ||
|
|
||
| # Mark a commit as release-certified by attaching a git note on the | ||
| # 'certification' notes ref and pushing it to origin. | ||
| # | ||
| # Usage: | ||
| # certify.sh <commit-sha> --run-url <url> [--suite <text>] | ||
| # | ||
| # The note body is a single-line JSON object with the commit, timestamp, | ||
| # workflow run URL and the certification suite summary. Re-running on an | ||
| # already-certified commit is a no-op (exit 0). | ||
|
|
||
| NOTES_REF="refs/notes/certification" | ||
|
|
||
| usage() { | ||
| echo "Usage: $0 <commit-sha> --run-url <url> [--suite <text>]" >&2 | ||
| exit 2 | ||
| } | ||
|
|
||
| SHA="" | ||
| RUN_URL="" | ||
| SUITE="nightly certification" | ||
|
|
||
| while [ $# -gt 0 ]; do | ||
| case "$1" in | ||
| --run-url) RUN_URL="$2"; shift 2 ;; | ||
| --suite) SUITE="$2"; shift 2 ;; | ||
| -h|--help) usage ;; | ||
| -*) echo "ERROR: unknown option: $1" >&2; exit 2 ;; | ||
| *) SHA="$1"; shift ;; | ||
| esac | ||
| done | ||
|
|
||
| if [ -z "$SHA" ]; then | ||
| usage | ||
| fi | ||
|
|
||
| if ! git rev-parse --verify --quiet "$SHA^{commit}" >/dev/null; then | ||
| echo "ERROR: not a commit: $SHA" >&2 | ||
| exit 1 | ||
| fi | ||
| SHA=$(git rev-parse "$SHA^{commit}") | ||
|
|
||
| # '+' allows non-fast-forward notes updates, which are routine for git notes. | ||
| # A missing notes ref on origin is normal (first certification); any other | ||
| # fetch failure must fail loudly, else the already-certified check runs | ||
| # against a stale ref. | ||
| if ! git fetch origin "+$NOTES_REF:$NOTES_REF" 2>/dev/null; then | ||
| # ls-remote exit 2 means the ref is absent on origin (normal on first | ||
| # certification); any other status is a transport failure and must fail | ||
| # loudly, else the already-certified check runs against a stale ref. | ||
| LS_RC=0 | ||
| git ls-remote --exit-code origin "$NOTES_REF" >/dev/null 2>&1 || LS_RC=$? | ||
| if [ "$LS_RC" -eq 2 ]; then | ||
| echo "No certification notes on origin yet" >&2 | ||
| else | ||
| echo "ERROR: failed to fetch $NOTES_REF from origin (ls-remote rc=$LS_RC)" >&2 | ||
| exit 1 | ||
| fi | ||
| fi | ||
|
|
||
| if git notes --ref=certification list 2>/dev/null | awk '{print $2}' | grep -qx "$SHA"; then | ||
| echo "Commit $SHA is already certified; nothing to do" | ||
| exit 0 | ||
| fi | ||
|
|
||
| CERTIFIED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ) | ||
| NOTE="{\"commit\":\"$SHA\",\"certified_at\":\"$CERTIFIED_AT\",\"run_url\":\"$RUN_URL\",\"suite\":\"$SUITE\"}" | ||
|
|
||
| # Runners carry no git identity; the note commit needs one. | ||
| git -c user.name="Datadog Java Profiler" -c user.email="java-profiler@datadoghq.com" \ | ||
| notes --ref=certification add -f -m "$NOTE" "$SHA" | ||
| git push origin "$NOTES_REF" | ||
|
|
||
| echo "Certified $SHA" | ||
| echo "$NOTE" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # Copyright 2026, Datadog, Inc | ||
|
|
||
| # File a Jira issue, or update the existing open one when a duplicate is | ||
| # already tracked. | ||
| # | ||
| # Environment: | ||
| # ATLASSIAN_BASE_URL e.g. https://datadoghq.atlassian.net (required) | ||
| # ATLASSIAN_USER_EMAIL account email for the API token (required) | ||
| # ATLASSIAN_API_TOKEN API token (required) | ||
| # JIRA_PROJECT project key, default PROF | ||
| # | ||
| # Usage: | ||
| # file-jira-issue.sh --summary <text> --body-file <path> [--priority <name>] | ||
| # [--label <label> ...] | ||
| # | ||
| # Dedupe: when an unresolved issue in the project already matches the | ||
| # summary text (JQL '~' is a text match, not exact equality), a dedupe | ||
| # comment is added to it instead of creating a new issue. A failed dedupe | ||
| # lookup only warns: a duplicate ticket beats a lost failure report. | ||
| # Prints the created/updated issue key on stdout. | ||
|
|
||
| usage() { | ||
| echo "Usage: $0 --summary <text> --body-file <path> [--priority <name>] [--label <label> ...]" >&2 | ||
| exit 2 | ||
| } | ||
|
|
||
| SUMMARY="" | ||
| BODY_FILE="" | ||
| PRIORITY="High" | ||
| LABELS=() | ||
|
|
||
| while [ $# -gt 0 ]; do | ||
| case "$1" in | ||
| --summary) SUMMARY="$2"; shift 2 ;; | ||
| --body-file) BODY_FILE="$2"; shift 2 ;; | ||
| --priority) PRIORITY="$2"; shift 2 ;; | ||
| --label) LABELS+=("$2"); shift 2 ;; | ||
| -h|--help) usage ;; | ||
| *) echo "ERROR: unknown argument: $1" >&2; usage ;; | ||
| esac | ||
| done | ||
|
|
||
| if [ -z "$SUMMARY" ] || [ -z "$BODY_FILE" ]; then | ||
| usage | ||
| fi | ||
| if [ ! -f "$BODY_FILE" ]; then | ||
| echo "ERROR: body file not found: $BODY_FILE" >&2 | ||
| exit 1 | ||
| fi | ||
| for var in ATLASSIAN_BASE_URL ATLASSIAN_USER_EMAIL ATLASSIAN_API_TOKEN; do | ||
| if [ -z "${!var:-}" ]; then | ||
| echo "WARNING: $var is not set; cannot file a Jira issue" >&2 | ||
| exit 3 | ||
| fi | ||
| done | ||
|
|
||
| PROJECT="${JIRA_PROJECT:-PROF}" | ||
| BASE="${ATLASSIAN_BASE_URL%/}" | ||
| AUTH="$ATLASSIAN_USER_EMAIL:$ATLASSIAN_API_TOKEN" | ||
| API="$BASE/rest/api/3" | ||
|
|
||
| # Escaped JQL for the dedupe lookup; summary text is plain ASCII produced by | ||
| # the callers (no quotes), but escape anyway. The /search endpoint is retired; | ||
| # /search/jql takes a POST body and may return issues or issueKeys. | ||
| JQL_SUMMARY=$(jq -rn --arg s "$SUMMARY" '$s | gsub("\\\\"; "\\\\\\\\") | gsub("\""; "\\\\\"")') | ||
| JQL="project = $PROJECT AND summary ~ \"$JQL_SUMMARY\" AND resolution = Unresolved" | ||
| EXISTING_KEY=$(curl -fsS -u "$AUTH" -H 'Content-Type: application/json' \ | ||
| -X POST --data "$(jq -n --arg jql "$JQL" '{jql: $jql, maxResults: 1, fields: ["key"]}')" \ | ||
| "$API/search/jql" | jq -r '.issues[0].key // .issueKeys[0] // empty') || { | ||
| echo "WARNING: dedupe lookup failed; creating a new issue" >&2 | ||
| EXISTING_KEY="" | ||
| } | ||
|
|
||
| BODY=$(cat "$BODY_FILE") | ||
|
|
||
| if [ -n "$EXISTING_KEY" ]; then | ||
| COMMENT_PAYLOAD=$(jq -n --arg body "$BODY" \ | ||
| '{body: {type: "doc", version: 1, content: [{type: "paragraph", content: [{type: "text", text: ("Recurring failure. Details:\n\n" + $body)}]}]}}') | ||
| curl -fsS -u "$AUTH" -H 'Content-Type: application/json' \ | ||
| -X POST --data "$COMMENT_PAYLOAD" "$API/issue/$EXISTING_KEY/comment" >/dev/null | ||
| echo "$EXISTING_KEY" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # ADF document: the body file holds plain text; blank lines split paragraphs. | ||
| ADF_DOC=$(jq -n --arg body "$BODY" ' | ||
| [$body | split("\n\n")[] | {type: "paragraph", content: [{type: "text", text: .}]}] | | ||
| {type: "doc", version: 1, content: .}') | ||
|
|
||
| ISSUE_PAYLOAD=$(jq -n --arg project "$PROJECT" --arg summary "$SUMMARY" --arg priority "$PRIORITY" \ | ||
| --argjson doc "$ADF_DOC" --argjson labels "$(printf '%s\n' "${LABELS[@]:-}" | jq -R . | jq -s 'map(select(length > 0))')" ' | ||
| {fields: { | ||
| project: {key: $project}, | ||
| summary: $summary, | ||
| description: $doc, | ||
| issuetype: {name: "Task"}, | ||
| priority: {name: $priority}, | ||
| labels: $labels | ||
| }}') | ||
|
|
||
| ISSUE_KEY=$(curl -fsS -u "$AUTH" -H 'Content-Type: application/json' \ | ||
| -X POST --data "$ISSUE_PAYLOAD" "$API/issue" | jq -r '.key') | ||
|
|
||
| echo "$ISSUE_KEY" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.