Skip to content

feat(cua): host Cua Driver on macOS, Linux, and Windows - #16527

Draft
juliusmarminge wants to merge 1 commit into
mainfrom
t3code/cua/1-host
Draft

juliusmarminge wants to merge 1 commit into
mainfrom
t3code/cua/1-host

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Supersedes https://git.cafe/pingdotgg/t3code/pulls/15006 and https://git.cafe/pingdotgg/t3code/pulls/10899. This stack (https://git.cafe/pingdotgg/t3code/pulls/16527 → https://git.cafe/pingdotgg/t3code/pulls/16528 → https://git.cafe/pingdotgg/t3code/pulls/16529 → https://git.cafe/pingdotgg/t3code/pulls/16530 → https://git.cafe/pingdotgg/t3code/pulls/16531 → https://git.cafe/pingdotgg/t3code/pulls/16532) ships computer use and agent browser tabs on macOS, Linux, and Windows together.

Computer use needs Cua Driver running on the environment's host. https://git.cafe/pingdotgg/t3code/pulls/15006 asked users to install it themselves and only worked on macOS; Cua's own https://git.cafe/pingdotgg/t3code/pulls/10899 bundled it but only wired macOS end to end.

This layer hosts the pinned Cua Driver 0.38.0 on macOS, Linux, and Windows. The @trycua/cua-driver SDK is on the same 0.38.0. The SDK only checks that the driver's contract, tool-schema, capability, and MCP protocol versions match, so a later SDK bump can trail a driver bump.

  • Desktop builds bundle it for every target: one universal macOS executable, and per-arch Linux and Windows bundles. Electron starts it as an embedded host, so macOS grants Accessibility and Screen Recording to T3 Code, not to a second app. It hands the MCP descriptor to its primary backend over the existing private control pipe.
  • A standalone server (npx t3) downloads the same release into T3's home on first use, checksum- and size-verified, with path-traversal checks on the archive. T3CODE_CUA_DRIVER_PATH still overrides it.
  • The release table (asset names, sizes, SHA-256) lives in @t3tools/shared/cuaDriverRelease, so the build script and the server cannot drift. @trycua/cua-driver moves to the pnpm catalog, so desktop and server share one version.

Nothing attaches the driver to agents yet; that is the next layer. The setting stays off by default.

Ported from https://git.cafe/pingdotgg/t3code/pulls/10899 by Francesco Bonacci, with Julius's Effect-convention fixes from https://git.cafe/pingdotgg/t3code/pulls/11709, rebased onto the V2 orchestrator.

  • The SDK loads in place from the app archive. A small patch to @ubjs/node (the SDK's native loader) points dlopen at the .asar.unpacked sibling, the same fix fff already carries. Only the SDK's platform packages are unpacked, so Windows stays under its 80-file installer budget.

Verification

  • Focused tests: CuaDriver (start, revoke, crash, bounded shutdown), CuaDriverInstallation (verified install, checksum rejection, unsafe archive), DesktopCuaDriver, desktop telemetry and backend tests, and build-script staging for all five archive layouts. Server, desktop, contracts, and shared typecheck clean.
  • Checked by hand on macOS arm64 with 0.34.0: Cua's EmbeddedCuaDriverHost starts, returns its MCP descriptor and socket, and listApps answers through it. 0.38.0 has not been rechecked by hand.
  • The archive checksums and sizes match Cua's published checksums.txt and assets for 0.38.0.
  • skips the primary native probe for cross-architecture Windows payloads in build-desktop-artifact.test.ts fails on main too, so this PR did not cause it.

Built with Claude Opus 5.5 in T3 Code, through the Claude Code harness.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the vouch:trusted Author is trusted by repo permissions or the VOUCHED list. label Oct 6, 2026
@juliusmarminge juliusmarminge changed the title t3code/cua/1 host feat(cua): host Cua Driver on macOS, Linux, and Windows Oct 6, 2026
@github-actions github-actions Bot added the size:XXL 1,000+ changed lines (additions + deletions). label Oct 6, 2026
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 7da318e · PR result: e11023b · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@juliusmarminge

Copy link
Copy Markdown
Member Author

Taking over the host-layer assessment for maintainer batch desktop-app-12. I will check whether it can be rebuilt from current main as focused distribution, host lifecycle, and desktop integration layers without expanding into the later agent-attachment PRs. The existing draft and native qualification gaps stay explicit; no merge is planned.

@juliusmarminge

Copy link
Copy Markdown
Member Author

Thanks @juliusmarminge for establishing the host lifecycle, native SDK/permission constraints, pinned release facts, and packaging work in this draft.

I rebuilt this against current main as a smaller review sequence, with fresh test and implementation agents and independent Opus/Sol reviews:

  • #21694: pinned SDK/release manifest and verified standalone installation.
  • #21695: opt-in server lifecycle, settings, and private desktop handoff.
  • #21696: Electron-owned hosting, permission restart, and primary-only private report routing.
  • #21697: verified release staging, native ASAR loading, and minimal runtime resources.

I compared the entire original diff against the stack. All host-only capabilities are accounted for. The knip allowlist changes were unnecessary with the rebuilt dependency graph. Unreachable System Settings permission branches were removed because the current contract only accepts full-disk-access; the real SDK permission prompt and restart IPC remain. Unused SDK libraries were removed from desktop binary resources and the Windows server sidecar. Standalone SDK installation remains supported. The original 39 added behavior cases became 32 focused cases, with real compressed-archive and sidecar packaging coverage.

Three full-stack review rounds are complete; the final round is clean. The rebuilt Linux download/install/acquire/MCP path, cleanup, single-executable loading, full CLI executable build, and actual headless Electron ASAR native loading have passed. The PR bodies distinguish these proofs from full packaged GUI qualification.

Signed macOS TCC/cursor behavior, installed Windows UIA/NSIS, and Linux GUI capture/input and GNOME extension activation still need human qualification. Packaging adds resources before there is a client entry point, so its body recommends landing it with the attachment feature. Later layers of the original series are outside this takeover and were not modified.

Closing this draft in favor of the linked replacement stack. Nothing was merged.

Implementation: Codex gpt-6.1-sol; tests: Codex gpt-6-astra; reviews: Claude claude-opus-5-5 and Codex gpt-6-sol. Harness: T3 Code delegated tasks.

@steve-cooks steve-cooks left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A first-use download in standalone mode is cut off after 30 seconds and starts over on every retry. Found by reading the code; I didn't run it.

CuaDriver.ts#L222

const host = yield* createHost.pipe(Effect.tapError(rememberFailure), Effect.timeout(START_TIMEOUT)); // "30 seconds"

On the standalone path (not desktop), createHost is makeStandaloneHostFactory(installation.executable, …). executable downloads and installs the release when it isn't installed yet, so the whole download runs inside the 30-second START_TIMEOUT, even though CuaDriverInstallation gives the download Effect.timeout("10 minutes"). When the timeout hits, the download is interrupted and its scoped staging directory (makeTempDirectoryScoped) is deleted, so the next attempt downloads the 34–53 MB archive from scratch. On any connection that can't finish within 30 seconds, npx t3 / headless hosts never get computer use. One fix would be to resolve and install the binary outside the start timeout, or to give installation its own timeout.

Reviewed with GPT-6.1-Sol xhigh in Codex, verified by Claude Opus 5.5 High in Claude Code, running in T3 Code.

kimkhoi2202 added a commit to kimkhoi2202/t3code that referenced this pull request Nov 14, 2026
Squash of Julius's stack pingdotgg#16527 -> pingdotgg#16532 plus pingdotgg#16975
(head 3f61c22), rebased onto custom.

Conflict resolutions against current main:
- Effect's crypto API moved: effect/Crypto -> effect/crypto/Digest and
  SecureRandom (CuaDriver, darwin/win32 icon sources, tests).
- New Effect lint rules: Effect.try/tryPromise get typed catch handlers;
  CuaDriver reads T3CODE_CUA_DRIVER_PATH through HostProcess.Environment.
- Codex: keep main's per-thread threadId and resumeNativeThread, add
  userCuaDriver; OpenCode: runtime instructions moved into
  turnPromptOptions; ACP keeps opt-in HTTP MCP plus local stdio servers.
- Native app icons: take the per-platform sources from the stack.
- Mobile: main's settings screen plus the Computer use toggle.
@github-actions

Copy link
Copy Markdown
Contributor

This PR changes the test suite by +37 test cases (browser 0, unit +14, server +23).

Keep tests that would fail for a real reason, and test logic below the UI where it can be. See Testing.

Desktop builds bundle the pinned Cua Driver 0.38.0 for every target (one
universal macOS executable, per-arch Linux and Windows bundles). Electron
starts it as an embedded host so macOS grants follow T3 Code, and hands the
MCP descriptor to its primary backend over the private control pipe.

A standalone server downloads the same checksum-pinned release into T3's
home on first use; T3CODE_CUA_DRIVER_PATH still overrides it. The release
table lives in @t3tools/shared so the build and the server cannot drift,
and the SDK moves to the pnpm catalog at the same 0.38.0.

Ported from #10899 by Francesco Bonacci, with Julius's convention fixes.

Co-Authored-By: Francesco Bonacci <f@trycua.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants