Skip to content

fix(check-affected): run the unit suite for every production source change - #4025

Open
thymikee wants to merge 4 commits into
mainfrom
fix/check-affected-full-unit
Open

thymikee wants to merge 4 commits into
mainfrom
fix/check-affected-full-unit

Conversation

@thymikee

@thymikee thymikee commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Summary

#3995 passed pnpm check:affected --run --head and then failed eager-closure-budgets in CI Coverage. vitest related follows imports, but that gate reads sources as text, so related never selects it. The runner also dropped unit whenever vitest-related ran, even for fail-open plans. Most source paths (src/, packages/daemon, packages/selectors) never selected unit at all. Other unit-core tests have the same blind spot, for example cli-startup-import-closure.test.ts, so a hand list or a location-based set would miss some.

  • A production change under src/ or packages/*/src now selects unit (own:unit-text-gates).
  • unit never defers to the related run. A fail-open plan runs every local check. provider-integration still defers on a narrowed plan.
  • Test-only changes are unchanged.

Cost: about 3 min per source-changing run (check:unit: 165 s vitest plus smoke).

Validation

  • pnpm check:affected --run --head 919d757732: 42 checks passed, exactCommit=true, including unit and provider-integration.
  • check:affected:test: 86 pass. Restoring the unit skip, the fail-open skip, or the model rule each fails the new tests.
  • End to end: a committed plant that added one module to the platform-apple runner closure passed vitest-related. unit then failed it on eager-closure-budgets (exactCommit=true). The plant was dropped.
  • Docs delta 919d757..02d0e69: agent-guidance passed (exactCommit=true).
  • Not device-facing.

🤖 Generated with Claude Code

…hange

`vitest related` follows imports, but unit-suite gates such as eager-closure-budgets read
production sources as text, so no related test reaches them. The runner also skipped `unit`
whenever the related run was selected, and that skip applied even to fail-open plans. #3995
passed `check:affected --run --head` locally and then failed the closure budget in CI.

A production source change under src/ or packages/*/src now selects `unit`, and the related run
no longer stands in for it. A fail-open plan runs every local check. Test-only changes keep the
related run alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
agent-device-website Ignored Ignored Preview Oct 28, 2026 4:03pm UTC

Request Review

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.56 MB 5.56 MB 0 B
Package (unpacked) 5.56 MB 5.56 MB 0 B
Package (download) 1.69 MB 1.69 MB +10 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 30.7 ms 29.2 ms -1.5 ms
CLI --help 56.5 ms 58.3 ms +1.8 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/check-affected/run.test.ts Outdated
…ider-integration

The narrowed-plan case passed even when provider-integration was never selected or the
related run never ran. It now asserts the plan selects provider-integration, the related run
executes, and the provider suite does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

The code looks good at 988c310, and I found no problems. Not blocking, and fine to take or leave: add one sentence under "Interpret selection" in docs/agents/testing.md saying that a production source change runs the full unit suite (about 3 minutes) because some gates read sources as text, and that provider-integration is the only check the related run stands in for. Also, the test "the related run still stands in for provider-integration" in scripts/check-affected/run.test.ts only asserts exit 0, so it would be stronger if it recorded the executed commands and asserted that a related command ran and no test:integration:provider command ran.

CI is still running. Repo Guards & Compatibility, Coverage and both Smoke Tests are in progress and none has failed. Repo Guards runs the affected-selector tests this PR edits, so a failure there would be related. A failure in Coverage or Smoke Tests would look unrelated, because the diff only touches scripts/check-affected. I did not run check:affected:test or the mutation experiments, so those claims rest on the PR description. I also did not list every unit test that reads sources as text. There are no known conflicts.

One open thread does not apply, so please resolve it: #4025 (comment) (if the related command were missing, provider-integration would run and the test would fail; the model test guards the provider selection).

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 28, 2026
Under "Interpret selection", state that related tests replace only provider integration and
that a production source change also runs the unit suite, because some unit gates read sources
as text. To stay within the agent-docs budget, three lists become sentences and two sentences
that repeat tool output are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

The change at 919d757 looks good to me, and the earlier review of 988c310 still holds. The provider-deferral test in scripts/check-affected/run.test.ts now captures the executed commands. It asserts that the related run ran, that test:integration:provider did not, and that the plan includes provider-integration. That test would no longer pass if both commands were missing, so the earlier thread is fixed: #4025 (comment). I judged this by reading run.ts and the test, and I did not run the suite locally. All 20 checks pass, and the latest change only touches a scripts test. There are no conflicts.

@thymikee

Copy link
Copy Markdown
Member Author

Both notes are applied:

  • testing.md (dbaeb81): "Interpret selection" now says related tests replace only provider integration locally. It also says a production source change runs the unit suite (about 3 minutes), because some unit gates read sources as text. To stay within the 40,000-byte agent-docs budget, three bullet lists became sentences. I also removed two sentences that repeat tool output: the Ctrl-C hint the waiter prints, and the receipt's not-run status. The total is now 39,981 bytes.
  • Provider test (919d757): It records the executed commands. It asserts that the plan selects provider-integration, that a related command runs, and that test:integration:provider does not. Removing the deferral fails it. The thread is resolved.

Validation:

  • The full gate at 919d757 passed 42 checks (exactCommit=true).
  • The docs-only delta 919d757..dbaeb81 passed agent-guidance (exactCommit=true).

Two full-gate attempts at dbaeb81 failed in unrelated suites, with the host at load average 48–55:

  • host-kit/src/internal/exec-kill-settle.test.ts "a cancelled command with a kill policy is signalled…" failed once. It passed 5/5 alone. The test aborts after a fixed sleep(100), so a shell that has not yet installed its trap is killed outright and never writes the marker.
  • provider-scenarios/android-alert.test.ts failed with 5 s timeouts. Alone it failed once cold under load, then passed twice.

This PR does not touch either test or its imports. Because the unit suite now runs locally on every production change, races like these will surface more often. The kill-settle test should wait for a ready marker instead of sleeping; that would be a separate host-kit PR.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/agents/testing.md Outdated
A fail-open plan runs every local check, so related tests stand in for nothing there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

The change looks good at 02d0e69, and the unit suite now runs for every production source change as intended. The only change since the earlier review at 919d757 (#4025 (comment)) is in docs/agents/testing.md. It now says that on a narrowed plan, related tests replace only provider integration, which matches the code. All 20 reported checks pass, and there are no conflicts.

Both earlier Cubic threads are fixed at this head, so you can resolve them: the test that could pass with both checks absent (#4025 (comment)) now asserts the related run happened and provider integration did not, and the testing.md wording (#4025 (comment)) matches the plan logic.

I read the head code and tests but did not run the unit suite or the check-affected tests locally. I also did not check the '(about 3 minutes)' unit-suite duration, which is unmeasured.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant