Skip to content

babysit-pr: poll CI through REST, back off idle polls (PROF-16822) - #1038

Open
jbachorik wants to merge 7 commits into
mainfrom
jb/babysit-pr-rest-poll
Open

jbachorik wants to merge 7 commits into
mainfrom
jb/babysit-pr-rest-poll

Conversation

@jbachorik

@jbachorik jbachorik commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:
babysit-pr polls an open PR every 10 minutes; each poll spent ~13 of the 5000 hourly GraphQL points, almost all from gh pr checks, which fetches the full CI check rollup through GraphQL (~300 contexts on a typical java-profiler PR). The poll step now reads every signal through the REST API (pulls/<n>, commits/<sha>/check-runs, commits/<sha>/status, issues/<n>/comments, pulls/<n>/reviews, pulls/<n>/comments), and idle polls back off from 10 to 20 minutes, resetting on any event. Babysitter GraphQL usage drops to zero; the shifted volume is ~8 REST requests per poll against the separate REST bucket.

Motivation:
Measured on gh 2.103.3 against PR #1035 (GH_DEBUG=api trace + /rate_limit before/after): gh pr checks is 6 requests, all GraphQL, 12 points per invocation; gh pr view --json adds 1. A full 2-hour run costs ~156 GraphQL points per babysitter — ~3% of the hourly quota each, which multiplies with concurrent babysitters and every other gh call sharing the token (PROF-16822).

The REST calls were verified live to cover every signal the loop reads: check names, status, conclusion, details URL, check suites for reruns, status contexts (bits-review/head, dd-gitlab/*), PR state, mergeable_state, draft, top-level comments, and review bodies. The skill also states the traps the old GraphQL command hid: REST state and check status values are lowercase (open; queued, in_progress, completed — comparing against the GraphQL-cased values would stop the babysitter on an open PR or read every check as pending), --paginate emits one JSON object per page, so each read takes --jq '.check_runs[]' / .statuses[]; a head carries cancelled runs of superseded retries beside their reruns, so checks are deduped by name keeping the newest started_at. The thread census stays scoped to the inline-comment list; top-level comments and review bodies feed step 4's classification directly, keeping the replied record on one id sequence.

Dependent references updated: step 3 keys on REST mergeable_state: dirty (lowercase) instead of GraphQL mergeStateStatus: DIRTY; step 5 takes blocking reviews from the REST pulls/<n>/reviews list the census already uses.

Additional Notes:

  • Review: sphinx review (standard effort, 5 reviewer streams + adversary) over the original diff — 1 of 2 findings accepted and fixed, 1 rebutted with the measured trace (it claimed gh pr checks is REST-based; the trace shows 6/6 GraphQL requests). The Bits review on this PR raised three REST-semantics findings (P1 lowercase open stop-guard; lost top-level comment/review sources; combined-status pagination) — all verified and fixed in bc6ee80/89a595764; the fix range was re-reviewed fresh-context (no Critical/High; two MEDIUM fixed in 89a5957, one MEDIUM + one LOW fixed in d2cb116; two LOW wording nits left, outcome-equivalent). Ticket: PROF-16822.
  • Human review: recommended only to sanity-read the rewritten poll paragraph; no code, no CI config, agent-behavior change only.
  • Accepted cost: an idle-poll backoff delays reaction to new comments by up to 10 extra minutes in the quiet case; red checks found at a 20-minute interval are handled within the same run caps.

How to test the change?:
No code change — the diff edits one skill's instructions. Validation was run against PR #1035: the REST calls reproduce gh pr checks output (268 check-runs deduped by name, 30 status contexts, mergeable_state: blocked on a blocked PR), and the same reads ran live against this PR's own CI. A future babysit run on this checkout exercises the loop directly.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16822

@jbachorik jbachorik added the AI label Nov 20, 2026
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The REST endpoint returns lowercase open, so the retained OPEN guard tells the babysitter to stop on an open PR. The rewritten poll also omits explicit top-level feedback polling and pagination of commit-status contexts.

Open Bits AI session

🤖 Bits Code Review · Commit 4a30bbf

Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
@dd-octo-sts

dd-octo-sts Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #45072103481 | Commit: 76746b8 | Duration: 21m 30s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-22 21:51:22 UTC

@jbachorik
jbachorik marked this pull request as ready for review November 20, 2026 18:06
@jbachorik
jbachorik requested a review from a team as a code owner November 20, 2026 18:06
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T18:08:00.186605Z 4a30bbf Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a30bbf818

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
Comment thread .agents/skills/babysit-pr/SKILL.md Resolved

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The REST conversion retains uppercase OPEN guards, causing the babysitter to stop on open PRs. It also truncates commit-status contexts and removes explicit polling of top-level review feedback.

Open Bits AI session

🤖 Bits Code Review · Commit 4a30bbf

Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
@jbachorik
jbachorik force-pushed the jb/babysit-pr-rest-poll branch from 4a30bbf to d2cb116 Compare November 20, 2026 18:24
@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The new failure rule classifies unfinished checks with null conclusions as red, potentially triggering failure handling and preventing idle backoff. A completion guard separates pending checks from failed checks.

Open Bits AI session

🤖 Bits Code Review · Commit d2cb116

Comment thread .agents/skills/babysit-pr/SKILL.md Resolved

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The specialist verified the REST reads against the reviewed head and returned no actionable findings. The full polling loop remains untested, and GraphQL permissions prevented comparison with gh pr checks.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit d2cb116

@dd-octo-sts

dd-octo-sts Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

❌ 38 passed, 0 failed, 2 without a result out of 40 expected configurations

Test Matrix

Platform JDK 8 JDK 11 JDK 17 JDK 21 JDK 25
glibc-x64-hotspot ❔ ✅ ✅ ✅ ✅
glibc-x64-openj9 ❔ ✅ ✅ ✅ ✅
glibc-arm64-hotspot ✅ ✅ ✅ ✅ ✅
glibc-arm64-openj9 ✅ ✅ ✅ ✅ ✅
musl-x64-hotspot ✅ ✅ ✅ ✅ ✅
musl-x64-openj9 ✅ ✅ ✅ ✅ ✅
musl-arm64-hotspot ✅ ✅ ✅ ✅ ✅
musl-arm64-openj9 ✅ ✅ ✅ ✅ ✅

No result produced

These configurations wrote no validation log, so their outcome is unknown and
they are counted as failures. The job usually timed out or failed during setup;
see its log in the pipeline.

  • glibc-x64-hotspot-jdk8
  • glibc-x64-openj9-jdk8

Links

@jbachorik jbachorik added the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
@jbachorik

jbachorik commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you.

This run
  • 19:59 · started · CI on head d2cb1169 green, bits-review/head PASS (review 5997557657)
  • 20:01 · answered · 6 threads — 5 already fixed on d2cb1169 (lowercase open guards ×2, status pagination ×2, top-level comment/review polling), 1 valid P2 on SKILL.md:148 (queued checks must stay pending, not red) flagged to the session — the skill is the session's meta file, never the worker's edit scope
  • 20:00 · stopped · CI green on d2cb1169, Bits PASS, no fixes pushed, no commits made

Commits pushed: none (all findings were either already fixed on the current head or outside the worker's edit scope). Findings flagged: P2 on .agents/skills/babysit-pr/SKILL.md:148 — queued check runs (conclusion null) fall into the red rule; fix belongs to the session (meta file). Threads left for the author: none — every trusted root answered; unresolved P2 flagged above is the session's work queue. bits_triggered: 4a30bbf818de22962f43dbcea431803d9401a063, d2cb1169bfbc482e537759d003a3d2f1027e929b (both seeded, none posted this run). bits_rerequested: none. Note: the PR is no longer a draft (draft: false).

✅ stopped 20:00 UTC · CI green on d2cb1169, Bits PASS, 6 threads answered, 1 P2 flagged to the session, 0 commits pushed

@jbachorik jbachorik removed the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
@jbachorik jbachorik added the dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates label Nov 22, 2026
@jbachorik

jbachorik commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you.

This run
  • 20:06 · started · CI on head a29b49a73 in progress, bits-review/head pending (PR non-draft, push should re-trigger Bits)
  • 20:26 · CI · poll clean — GitHub Actions nearly green (test-matrix still running), dd-gitlab/* mostly green (build-artifact, fuzz-build-and-soak, default-pipeline pending), bits-review/head pending
  • 20:47 · CI red · test-matrix green but dd-gitlab/integration-test-x64-glibc hotspot-8 and openj9-8 failed on a flaky JDK8 download (release-asset URL served an S3 error body, 3/3 attempts) — retrying both jobs once
  • 20:49 · retried · integration-test-x64-glibc: [hotspot, 8] (2287221321) and …[openj9, 8] (2287221327) queued; dd-gitlab/post-pr-comment failed only downstream (2 configs without a result)
  • 21:04 · CI red · both retries failed identically (same JDK8 URL returns a 1867-byte XML error body, deterministic) — main pipelines fail the same way since ~06:00 UTC; external infra outage, flagged to the operator (fix would be in .gitlab/ CI config — outside the workers scope)
  • 21:12 · Bits · @datadog review posted for a29b49a73 (gate pending since the push; operator approved) — once-per-head recorded
  • 21:20 · Bits · review returned FAIL on a29b49a73 — one finding: check-run dedupe by newest started_at can discard a queued replacement run (stale results drive remediation or a premature stop) — fix belongs to .agents/skills/babysit-pr/SKILL.md, the sessions meta file; flagged, no worker reply possible on a review body
  • 21:21 · stopped · blocked on CI infra (JDK8 download outage; retries exhausted, also red on main)

Commits pushed: none — nothing inside the workers edit scope broke; the only red is the external dd-gitlab JDK8-download outage (deterministic Adoptium S3 error body, 3/3 attempts in-job, one retry per job burned, same failure on main since ~06:00 UTC). Findings flagged: (1) Bits FAIL on a29b49a73 — SKILL.md check-run dedupe by newest started_at can discard a queued replacement (stale CI results drive remediation or a premature stop); fix belongs to the session (meta file). (2) Persistent JDK8 download outage in .gitlab/ tooling — sessions CI config. Threads left for the author: none — all 9 trusted inline roots were answered by previous runs; the new Bits finding arrived as a review body (no inline thread to answer). bits_triggered: 4a30bbf818de22962f43dbcea431803d9401a063, d2cb1169bfbc482e537759d003a3d2f1027e929b, a29b49a730eb75f6c6d553989d1520646c4c2fb6 (posted 21:12 for the third). bits_rerequested: none.

✅ stopped 21:21 UTC · blocked on CI infra (JDK8 download outage, retries exhausted, also red on main) · 0 commits pushed · 1 Bits finding flagged to the session

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 3 Pipeline jobs failed

DataDog/java-profiler | integration-test-x64-glibc: [hotspot, 8]

View more details · View in GitLab

DataDog/java-profiler | integration-test-x64-glibc: [openj9, 8]

View more details · View in GitLab

DataDog/java-profiler | post-pr-comment

View more details · View in GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 47cd7b4 | Docs | View more details | Give us feedback!

@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Deduplicating checks by started_at can discard a queued replacement, allowing stale CI results to drive remediation or a premature stop.

Open Bits AI session

🤖 Bits Code Review · Commit a29b49a

Comment thread .agents/skills/babysit-pr/SKILL.md Resolved
@jbachorik jbachorik added dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates and removed dog-sitter Actively watched by babysit-pr: fixes pushed through the repo gates labels Nov 22, 2026
@jbachorik

jbachorik commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you.

This run
  • 21:28 · started · CI on head 47cd7b4ab in progress; bits-review/head pending (auto-retriggered by the push)
  • 21:32 · answered · Bits P2 thread on SKILL.md dedupe wording (root 4660041761) — harm premise refuted, wording nit flagged to the session (skill file, never worker-edited)

✅ stopped 22:21 UTC · blocked on CI infra (JDK8 download outage) — all checks green except the known-outage aggregators; Bits PASS on 47cd7b4

  • 22:10 · CI red · only the known JDK8-download-outage signature: dd-gitlab/post-pr-comment, dd-gitlab/default-pipeline, DDCI Status (downstream aggregators) — no rerun, persistent upstream outage
  • 22:11 · Bits · re-triggered via top-level @datadog review — pending 45 min on green-except-outage head 47cd7b4
  • 22:14 · Bits · PASS on 47cd7b4 — bits-review/head green after the re-trigger
  • 22:21 · stopped · blocked on CI infra (JDK8 download outage) — only reds are the known-outage aggregators

Stop summary — no commits pushed this run (no fixable CI failures; the only reds are the persistent JDK8-download infra outage). Findings answered: 1 — Bits P2 thread on SKILL.md dedupe wording (root 4660041761), harm premise refuted on the thread; the residual wording nit is flagged to the session owner (skill file, never worker-edited). Threads left for the author: that SKILL.md wording nit (meta-file fix is the session's call). bits_triggered: 4a30bbf, d2cb116, a29b49a, 47cd7b4 (recorded after the 22:11 @datadog review re-trigger). bits_rerequested: none. Note: the dog-sitter label is missing from the PR — flagged, not created.

@jbachorik

Copy link
Copy Markdown
Collaborator Author

@DataDog review

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The REST polling instructions preserve pending-check handling and all three comment and review sources. The completed static review found no actionable regression in the supplied diff.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 47cd7b4

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant