Skip to content

ci(evals): Post each label's products matrix to Slack - #20274

Open
jaydgoss wants to merge 1 commit into
masterfrom
jaygoss/ai-evals-post-products-matrix
Open

ci(evals): Post each label's products matrix to Slack#20274
jaydgoss wants to merge 1 commit into
masterfrom
jaygoss/ai-evals-post-products-matrix

Conversation

@jaydgoss

@jaydgoss jaydgoss commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

DESCRIBE YOUR PR

Adds a report job to the AI Evals workflow that posts each label's products matrix (test x target x product, base and head) to Slack once the eval jobs finish, using the post-matrix action from sentry-for-ai-evals#111. This repo is public, so the matrix can't go on the PR. VDY-309

  • Same job and condition as getsentry/sentry-for-ai#385 and sentry-for-ai-evals#115. It runs only when the eval jobs succeeded or failed, so not when a new push cancelled them, when there's no preview, or for fork PRs.
  • The log shows only whether each label was posted. The channel is set in sentry-for-ai-evals, so this repo doesn't name it.
  • Posts with the EVALS_SLACK_BOT_TOKEN secret, which is set on this repo. Without it, the job logs a notice and posts nothing.

Testing: ai-evals:fastapi:sonnet-5.5 on this PR posted its matrix to Slack (run).

IS YOUR CHANGE URGENT?

Help us prioritize incoming PRs by letting us know when the change needs to go live.
Select exactly one option. For deadlines, replace YYYY-MM-DD with the due date. You can update this information later by editing the PR description.

  • Urgent deadline (GA date, etc.): YYYY-MM-DD
  • Other deadline: YYYY-MM-DD
  • No deadline: Not urgent, can wait up to 1 week+

SLA

  • Teamwork makes the dream work, so please add a reviewer to your PRs.
  • Please give the docs team up to 1 week to review your PR unless you've supplied a deadline.

Thanks in advance for your help!

PRE-MERGE CHECKLIST

Make sure you've checked the following before merging your changes:

  • Checked Vercel preview for correctness, including links
  • PR was reviewed and approved by any necessary SMEs (subject matter experts)
  • PR was reviewed and approved by a member of the Sentry docs team

🤖 Generated with Claude Code

@jaydgoss
ci(evals): Post each label's products matrix to Slack
ca26660 
A PR run's test x target x product matrix could only be seen by downloading both runs
from evals.sentry.dev and running products_matrix.py by hand, and this repo is public,
so it can't go on the PR. A report job now runs after the eval jobs and posts each
label's matrix to Slack as the sentry-evals bot, with sentry-for-ai-evals' post-matrix
action, the same job sentry-for-ai has. Its log shows only whether each label was
posted, and the channel is set in sentry-for-ai-evals, so neither the workflow nor the
log names it.

Refs VDY-309
@vercel

vercel Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
develop-docs Ready Ready Preview Oct 27, 2026 8:28pm UTC
sentry-docs Ready Ready Preview Oct 27, 2026 8:28pm UTC

Request Review

@github-actions github-actions Bot added the Priority: Normal Docs review has no urgent deadline label Oct 27, 2026
@jaydgoss jaydgoss added the ai-evals:llms_txt:sonnet-5.5:x3 Run the llms.txt eval 3 times on Sonnet 5.5, on this PR's preview and its merge base's docs label Oct 27, 2026
@sentry-evals

sentry-evals Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown

Sentry Evals

sentry-for-ai-evals:llms_txt:sonnet-5.5:x3 (Comparison)

Run Status Passed Failed Errored Cost Tokens Duration
Head Error 0 0 3 $0.00 0 0.00s
Base Error 0 0 3 $0.00 0 0.00s
Diff 0 0 0 $0.00 0 0.00s

@jaydgoss jaydgoss added ai-evals:fastapi:sonnet-5.5 and removed ai-evals:llms_txt:sonnet-5.5:x3 Run the llms.txt eval 3 times on Sonnet 5.5, on this PR's preview and its merge base's docs labels Oct 27, 2026
@sentry-evals

sentry-evals Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown

Sentry Evals

sentry-for-ai-evals:fastapi:sonnet-5.5 (Comparison)

Run Status Passed Failed Errored Cost Tokens Duration
Head Passed 1 0 0 $0.46 2.2M 134.81s
Base Passed 1 0 0 $0.44 1.5M 77.55s
Diff 0 0 0 +$0.01 (+2.7%) 🔴 +619.2k (+40.3%) 🔴 +57.26s (+73.8%)

@jaydgoss
jaydgoss marked this pull request as ready for review October 27, 2026 22:37
@jaydgoss
jaydgoss requested review from a team and evanpurkhiser October 27, 2026 22:37
@jaydgoss jaydgoss assigned Abdkhan14 and unassigned Abdkhan14 Oct 27, 2026
@jaydgoss
jaydgoss requested a review from Abdkhan14 October 27, 2026 22:37
@cursor

cursor Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The plan checks Slack posts of each label's products matrix, the report job result, and whether eval jobs still finish. An issue escalates when a finished eval run has no Slack post or the report job fails.

Not monitored: no services

Mention @change-monitor in a comment to update the plan.

Plan

What changed

Labeled pull request eval jobs that succeed or fail now start a report job. That job posts each label's products matrix to Slack.

No catalog service ships this change. GitHub Actions reads the workflow from the branch that runs it. The change does not go live through a docs site deploy.

Risk

If the report job fails, the AI Evals workflow fails. Reviewers may treat finished evals as failed.

If the Slack bot token is missing, the job logs a notice and posts nothing. Teams then miss the products matrix.

The new job does not change the eval job steps.

Intended effect

Each finished label should have a Slack post of its products matrix. Absent means eval jobs finished and that label has no Slack post.

Connected telemetry does not include Slack or GitHub Actions. There is no baseline to record here.

Regression watch

This change can fail the workflow after evals finish. It can also skip the Slack post. Those are the user-visible faults.

Connected telemetry does not include GitHub Actions job results. There is no measured baseline for those faults.

Not observable

Slack posts of each label's products matrix. There is no Slack connector.

GitHub Actions report job results, and log lines that say whether each label was posted.

The pre-ship rate of labeled eval runs that finish. There is no Actions history in connected telemetry.

Get Slack DMs about this change: cursor.com/automations/rollouts

@jaydgoss
jaydgoss requested review from a team and sfanahata October 27, 2026 22:38

This branch was successfully deployed

2 active deployments
Preview – sentry-docs — ca26660b Deployed Oct 27, 2026 by vercel[bot]
View deployment
Preview – develop-docs — ca26660b Deployed Oct 27, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@evanpurkhiser evanpurkhiser Awaiting requested review from evanpurkhiser
@Abdkhan14 Abdkhan14 Awaiting requested review from Abdkhan14
@sfanahata sfanahata Awaiting requested review from sfanahata

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Priority: Normal Docs review has no urgent deadline

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jaydgoss @Abdkhan14