Skip to content

feat(ci): enable engineer bot mention requests - #511

Merged
vuanhphung merged 12 commits into
mainfrom
feat/engineer-bot-mention
Oct 22, 2026
Merged

vuanhphung merged 12 commits into
mainfrom
feat/engineer-bot-mention

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Allow maintainers with write access to invoke the engineer bot with a top-level @bot comment on an open, same-repository PR. Review requests inspect the diff without edits; change requests can push fixes and reply. Mention and follow-up writers share a queue of up to 100 pending requests. Eligibility is rechecked before PR checkout. Prompts and cache warming use the trusted default branch; PR diffs are separate read-only reference files, with the base refreshed after queueing and the checked-out head SHA included in replies. The trigger activates once the workflow lands on main.

This PR includes merged #512: all six bot workflows use databricks-claude-opus-5-5, with engine commit ac3b08eab993ef14bf997a9bf4c776fa5acb16d8 from databricks/databricks-bot-engine#166. SDK/CLI defaults are 0.2.167 / 2.1.300, with Node 22 for the npm CLI. The engine adds [1m] automatically.

Validation: 39 local workflow scenarios and shell checks passed, including permission lookup failures, closed/merged PRs, and head changes after queueing. actionlint 1.7.12 passed with a temporary compatibility filter for the new queue key; queue: max was verified against GitHub's current syntax documentation. Diff checks passed.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Looks solid overall — the two-job authorize→mention split keeps secrets behind a write-access + non-fork check, reuses the trusted default-branch .bot, and the shared engineer-bot-comment-pr-<N> concurrency group correctly serializes with the follow-up workflow. One medium reliability concern: the local PR-diff step can fail on PRs whose base branch has moved because base.sha isn't fetched into the branch checkout.

Comment thread .github/workflows/engineer-bot-mention.yml
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
vuanhphung added a commit that referenced this pull request Oct 22, 2026
Configure all six bot workflows to use `databricks-claude-opus-5-5`
through the supported per-bot overrides. Pin the shared engine upgrade
and align the local SDK/CLI defaults to `0.2.167` / `2.1.300`, with Node
22 for the npm CLI. The engine adds the `[1m]` suffix automatically.

Pins merged databricks/databricks-bot-engine#166
(`ac3b08eab993ef14bf997a9bf4c776fa5acb16d8`). This PR is also stacked on
#511.

Validation: all six workflow callers, action YAML/shell syntax, and
engine/SDK/CLI/Node pin consistency passed. The engine upgrade passed
CI, including the JFrog SDK install and bundled CLI version check, plus
1,095 local tests (14 skipped).

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 3 Low

Looks reasonable overall, with 3 low-severity hardening/UX notes on the new mention workflow. Authorization (write-permission API check, same-repo/open PR check, User-type comment) and concurrency sharing with the follow-up workflow look correct. I couldn't verify engine-side flags (--phase mention, --bot) or the bumped SDK/CLI versions from this repo.

Comment thread .github/workflows/engineer-bot-mention.yml
Comment thread .github/workflows/engineer-bot-mention.yml Outdated
Comment thread .github/workflows/engineer-bot-mention.yml
@vuanhphung
vuanhphung force-pushed the feat/engineer-bot-mention branch from 85239b4 to 89fd6cc Compare October 22, 2026 18:10

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

The mention workflow mostly follows the existing follow-up workflow's security pattern: default-branch bot setup, a fork check, a write-permission gate, and a JFrog scrub. One medium issue: the shared concurrency group can silently drop queued requests. One low hardening point: the untrusted diff is concatenated into the system prompt. Nit, scope: the PR also moves every bot workflow to databricks-claude-opus-5-5 via new *_MODEL_OVERRIDE vars, and bumps Node 20→22 and the SDK/CLI versions. The description doesn't mention any of this, so please call it out for reviewers.

Comment thread .github/workflows/engineer-bot-mention.yml
Comment thread .github/workflows/engineer-bot-mention.yml Outdated
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good overall, with 2 low-severity concerns about how the mention workflow builds the diff and its offline module cache. The security setup holds up: write access is checked before any secrets are requested, the PR must come from this repo, bot setup and caches come from the default branch, and JFrog/OIDC credentials are scrubbed before the agent runs.

Comment thread .github/workflows/engineer-bot-mention.yml
Comment thread .github/workflows/engineer-bot-mention.yml
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low concern about the PR head moving while a request waits in the queue. The security setup is sound: write-access check before any secrets are used, same-repo PRs only, prompts and tool binaries from the trusted default branch, JFrog/OIDC credentials removed before the agent runs, and the diff reference file created exclusively, made read-only and git-excluded. The model, engine and SDK/CLI version bumps are applied consistently across all six workflows.

Comment thread .github/workflows/engineer-bot-mention.yml
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity concern: the PR state isn't re-checked when a queued request finally runs. The setup is well isolated: prompts and cache warming come from the default branch, tools come from the trusted checkout, bin/ and test-results/ are gitignored, and the trust model matches the existing follow-up workflow. queue: max and the engine's --phase mention support at the pinned SHA could not be verified from this checkout.

Comment thread .github/workflows/engineer-bot-mention.yml
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity robustness issue. The trust boundaries are handled well: the bot loads its setup from the default branch, removes registry credentials before the agent runs, rechecks the PR after queueing, keeps the diff as a read-only file outside git, and copies tool binaries with symlink checks.

Comment thread .github/workflows/engineer-bot-mention.yml
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good overall, with two low-severity concerns. The mention workflow gates on write access, runs only on same-repo PRs, loads its prompts, tools and cache from the trusted default branch, and removes the JFrog credentials before the agent runs. The model, engine and SDK bumps are applied the same way across all six workflows.

echo "ACTIONS_ID_TOKEN_REQUEST_TOKEN="
echo "ACTIONS_ID_TOKEN_REQUEST_URL="
} >> "$GITHUB_ENV"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — Recheck PR eligibility re-reads the PR's state, head repo and head ref after the queue wait, but it doesn't re-check the commenter's write permission. Requests can sit in the shared queue for a long time (up to 100 pending, 30–45 min each), so a request from someone whose access was revoked meanwhile will still run with the App token and model secrets. The PR description says "Eligibility is rechecked before PR checkout", which suggests the full gate. Consider repeating the collaborators/$COMMENTER/permission lookup here (pass github.event.comment.user.login through) and failing closed, as authorize does. Separately, the .head.ref == $ref comparison is effectively a no-op because GitHub doesn't allow a PR's head branch to change.

Comment thread .github/workflows/engineer-bot-followup.yml
Comment thread .github/workflows/engineer-bot-learning.yml
@vuanhphung
vuanhphung enabled auto-merge October 22, 2026 19:56
@vuanhphung
vuanhphung added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit d9a87e0 Oct 22, 2026
20 of 21 checks passed
@vuanhphung
vuanhphung deleted the feat/engineer-bot-mention branch October 22, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants