Skip to content

fix(workspace): report opening progress and preserve failed workspaces - #149

Open
AlexXPZhu wants to merge 8 commits into
Tencent:mainfrom
AlexXPZhu:fix/workspace-open-progress
Open

AlexXPZhu wants to merge 8 commits into
Tencent:mainfrom
AlexXPZhu:fix/workspace-open-progress

Conversation

@AlexXPZhu

@AlexXPZhu AlexXPZhu commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Fixes #124.

Slow sandbox/image builds left chat workspaces at “opening”, with no useful logs or problems. Opening now reports its current phase, elapsed time, timeout and bounded command output. Legacy workspaces with platform-managed repositories also report clone and branch progress, including Git transfer speed. Failures appear in chat, run logs and runs_get.problems, even when the workspace has no SOP nodes.

  • Feed workspace progress from the existing procwatch observer. Clone and Docker build use one execution/cancellation path, preserving stall detection, command timeouts, always-on clone progress, and tail-only errors from fix(workspace): report progress and stop hung clones and image builds (#124) #150. Existing run-event and runner-log observers remain active. The same Git execution path preserves upstream's managed SSH credentials, URL redaction and SSH branch safeguards.
  • Add configurable opening-phase deadlines and request-scoped runner progress/cancellation. Materialize lazy runner Docker sandboxes within the build phase (15 minutes by default), then start a separate initialization deadline (5 minutes by default). Preserve the silent --reference --dissociate repack exception and stop fallback retries after a stall or timeout. Bound unterminated output and redact complete lines before publishing them.
  • Retain the workspace binding and existing files after preparation failures. Calling workbench_workspace_open without arguments retries the same environment and run. New conversation workspaces stay empty until the agent performs its own Git operations, following feat(workspaces): open conversation workspaces by sandbox with agent-owned Git #947. For legacy platform-managed checkouts, recover only empty interrupted clones, preserve local work, and avoid repeating a full clone when no reference cache was used.
  • Show when the project's Dockerfile differs from the workspace's frozen version. Reopening the same environment preserves that version; a new conversation can use the updated environment. Document timeout settings, Git proxies, package mirrors and existing reference-cache support.
  • Keep polling responses small by retaining only the latest opening payload alongside ordinary lifecycle events. Reconcile those events so failures appear once and successful retries clear stale problems.

Scope

  • Platform core / API / runner
  • Public adapter / deployment
  • Classic
  • Portable
  • Shared behavior contract
  • Installer / CLI
  • Documentation only

Includes the native chat UI and deployment documentation.

Compatibility

Follow main's sandbox-only opening API: the only optional argument is sandboxProfileId. Choosing another environment switches to that environment's workspace; repeating the current choice reuses it. Agent-owned native CLI Git commands are outside the platform clone/branch deadline settings. Upstream's legacy Git-config restoration and removal of the old script helper are preserved.

No database migration: progress is stored in the existing run-event JSON payload and log tables. Workspace scheduler status remains waiting; readiness/failure is recorded separately.

Runner progress and cancellation use additive workspace.progress / workspace.cancel control messages. Upgrade both platform and runner for live progress and cancellation; older runners still return their final result and retain their existing preparation timeout. Ordinary runner checkouts retain main's configurable command/branch bounds (AGENTFLOW_GIT_CLONE_TIMEOUT, 10 minutes by default). Opening-phase and command deadlines both apply; documentation explains their precedence.

Retries preserve frozen environments and existing files. This does not add a shared credential-independent mirror cache or resume partially downloaded Git packs; completed checkouts, Docker layers and the existing local --reference --dissociate path are reused.

Validation

Validated merge commit a9d9ef8 against upstream main at 777f16a.

  • Relevant Go package suites passed: runner, engine, runtime, gitwork, procwatch, workspaceprep, domain, repo, httpapi and web, using CI's five real-browser test exclusions and 600-second package timeout.
  • Focused regressions passed for opening progress, cancellation, persistent CLI lifetime, same-workspace retry, and independent Docker build/initialization deadlines. The retry test verifies that new conversation workspaces do not auto-checkout project repositories and that a successful retry clears the previous failure.
  • The Docker deadline fixture now exercises direct script execution and the current workspace-mount inspection contract, without the removed script helper.
  • Chatbot: 374 tests passed; production build passed.
  • Passed go vet ./..., python3 scripts/check-public.py and bash scripts/scan-secrets.sh.

Docker deadline tests use controlled command fixtures. Real Docker builds, live model services, real-browser acceptance and the race detector were not rerun for this update. Classic/Portable-specific changes are inherited unchanged from upstream.

Safety

  • I did not include credentials, internal endpoints, production data, or organization-specific infrastructure.
  • I preserved user files and unrelated changes.
  • I updated generated Classic host bundles when applicable. — Not applicable.
  • I documented third-party content and licensing changes when applicable. — No new third-party code/assets or dependencies.

Implementation and tests prepared with Codex assistance.

Robert0622
Robert0622 previously approved these changes Nov 10, 2026

@Robert0622 Robert0622 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 050eb6c and exercised workspace-opening progress, failure preservation and retry behavior locally.

Validation on Linux:

  • Built this head's chatbot bundle; its workspace-opening component tests passed.
  • Clone/checkout, conversation workspace, preparation, runner control/progress and workspace-opening tests passed in gitwork, engine, runtime, runner, httpapi and repo.
  • All workspaceprep tests passed, including streaming before command exit, process-group cancellation, split-secret redaction, bounded unterminated output and timeout configuration.

I found no blocking issue in this diff. This was local subprocess/API/component validation; I did not simulate a real throttled network or use a live Docker daemon. Please coordinate with #150, which changes overlapping clone/build paths. LGTM on this head.

@Robert0622

Copy link
Copy Markdown
Collaborator

Thanks — the review and validation of 050eb6c still stand, and all 14 CI checks passed on that head. However, #150 has just been merged and it rewrote the same clone / docker build paths, so this PR now conflicts with main in:

  • internal/engine/repository_workspace.go
  • internal/gitwork/git.go
  • internal/runner/daemon.go
  • internal/runtime/sandbox_docker.go

This isn't a mechanical conflict: main now runs git clone / docker build through procwatch (stall + timeout watching, --progress always on, tail-only errors), while this PR routes the same commands through workspaceprep.Output for progress reporting. Could you rebase onto main and integrate the two — e.g. feed workspaceprep progress from a procwatch observer rather than running a second exec wrapper — keeping both the stall detection from #150 and the progress/failed-workspace preservation from this PR?

Once it's rebased and CI is green, I'll re-review the delta and merge.

@Robert0622

Copy link
Copy Markdown
Collaborator

LoopForge PR Review — head 0e2de75ba811388bbe07ef6d7c9d425ce3d53256
Verdict: ✅ Pass

  • Diff: 38 files, +1752/-228 across chatbot (new WorkspaceOpening.jsx + CSS + test), internal/workspaceprep (new progress/observer plumbing with URL-credential / auth-header / token-query redaction), internal/engine (new workspace_opening.go, repository_workspace.go binding-retention logic), internal/gitwork (incomplete_clone.go, token-scoped gitwork.C/WithToken), internal/runner (daemon.go, workspace_preparation.go, checkout helpers wrapped in workspaceprep.Do phases), internal/httpapi (assistant_run_view.go problem reconciliation, assistant_workspace_open.go retry/retain-binding path, runner_ws.go), internal/procwatch/internal/runtime (observer/progress wiring), docs/deployment.md.
  • GitHub Actions had only completed the opensource-scan job on this SHA at review time (other required checks hadn't started), so verification relied on an isolated local run (no secrets, no prod access):
    • Installed a Go 1.26.10 toolchain and Node 22 in the sandbox, built the chatbot (scripts/build-chatbot.sh): 47 test files / 294 tests passed, Vite build succeeded.
    • go test ./internal/engine/... ./internal/workspaceprep/... ./internal/procwatch/... ./internal/runner/... ./internal/httpapi/... ./internal/gitwork/... ./internal/repo/... ./internal/domain/... — all packages ok, including the new workspace_opening_test.go, observer_test.go, progress_test.go.
    • go test ./internal/runtime/... has 7 failures (TestDockerSandboxRunsOnDemandClaudeCodeFromMountedDirectory, TestDockerDiscoveryImagesAndBuildHelpers, TestDockerMasksAndManagerPaths, TestIsolatedRuntimeImageReusesInstalledImage, TestMaterializeSkillBinaryFiles, TestHostInsteadOfUsernamePromptThenAskPass, TestAgentStateMountsAndDockerManagerReclaim) that reproduce identically on base main (same sandbox lacks Docker-in-Docker and capability-drop privileges) — pre-existing sandbox-infra limitations, not caused by this PR.
  • Read the core logic: workspaceprep.Do/Report/Result/Sanitize scopes a per-phase deadline and redacts URL credentials/auth headers/token query params with UTF-8-safe output truncation before anything reaches the run timeline or chat. assistant_workspace_open.go now distinguishes a missing-branch error (still discards the workspace) from other preparation failures (keeps the binding and gives a retry instruction), matching the stated goal of retaining the binding for retries. assistantRunProblems/workspacePreparationProblems reconcile workspace.opening.* events against the older stage-completed events so a successful retry clears a stale failure while a later distinct failure still surfaces — covered by TestWorkspaceOpeningProblemsReconcileCommandEvents.
  • No blocking defects found in the reviewed paths. Note: CI other than the opensource scan had not started on this SHA at review time; worth checking once it runs.

@Robert0622

Copy link
Copy Markdown
Collaborator

LoopForge PR Review - commit 0e2de75

Verdict: Pass

GitHub Actions for this PR currently show action_required for e2e, platform, validate (workflow approval gate, likely fork/first-run), so there is no authoritative CI signal yet. I checked out this exact SHA in an isolated sandbox (no credentials, no production access) and ran the affected packages directly:

  • go test ./internal/gitwork/... ./internal/procwatch/... ./internal/workspaceprep/... -> all pass
  • go test ./internal/engine/... -> pass (31.8s)
  • go test ./internal/runner/... -> pass (31.3s)
  • go test ./internal/httpapi/... -run Workspace|Preparation -> pass (74.5s), covering workspace_opening_test.go and assistant_run_view_preparation_test.go
  • go test ./internal/runtime/... -> 7 failures when run as a full package; re-ran each failing test in isolation:
    • TestDockerSandboxRunsOnDemandClaudeCodeFromMountedDirectory, TestDockerDiscoveryImagesAndBuildHelpers, TestDockerMasksAndManagerPaths, TestIsolatedRuntimeImageReusesInstalledImage, TestAgentStateMountsAndDockerManagerReclaim: pass in isolation, and reproduce the same 5 failures on unmodified main (c87694a) when run as a full package. Matches this PR description exactly - pre-existing cross-test interference, not introduced by this PR.
    • TestMaterializeSkillBinaryFiles (exec bit stripped) and TestHostInsteadOfUsernamePromptThenAskPass (git askpass) also fail identically on main in this sandbox (root user strips exec bits / sandbox git networking) - sandbox-environment artifacts, not attributable to this PR.
  • go vet ./internal/... -> clean
  • python3 scripts/check-public.py -> "Public content check passed"
  • bash scripts/scan-secrets.sh -> could not run locally: gitleaks tarball extraction is blocked by the sandbox uid mapping (Cannot change ownership to uid 501), unrelated to this PR
  • Chatbot (294 tests) and platform console (webtest, 151 tests) were not re-run locally in this pass (no Node toolchain set up); relying on the PR authors reported results for those two.

No correctness issues found in the reviewed packages; scope (workspace-opening progress reporting, procwatch observer reuse, retained bindings on failed opens, base-branch validation) matches the stated design and is backed by passing tests. This is a large diff (36 files, ~2.8k diff lines) - this review focused on the new/changed logic paths and their direct test coverage rather than a line-by-line read of every file changed.

FYI: once a maintainer approves the pending workflow runs, worth a quick glance to confirm CI comes back green too.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply the build timeout to the deferred Docker image build

At lines 107–109, EnsureWorkspacePreparation runs under the prepare phase, but for Docker sandboxes this is also where the lazy image build is triggered. The earlier build phase only acquires/wraps the sandbox; it does not execute the image build. As a result, a cold build is cancelled by AGENTFLOW_WORKSPACE_PREPARE_TIMEOUT (5 minutes by default), even though AGENTFLOW_WORKSPACE_BUILD_TIMEOUT allows 15 minutes. A healthy build taking 5–15 minutes therefore fails prematurely.

I reproduced this through prepareRunnerWorkspace with a controlled Docker command fixture: with build timeout 4s and prepare timeout 1s, the deferred build starts and is cancelled after about 1s. Please put the actual image build under the build-phase deadline, retain the prepare deadline for subsequent initialization, and add a regression test covering a deferred build with a shorter prepare timeout.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for finding this. Fixed in 856fa6d. Lazy Docker startup now runs under the build-phase deadline, while initialization keeps its separate timeout. Added regression tests covering the cold-build scenario and both timeout paths; they reproduce the issue before the fix and pass afterward. Could you take another look?

@Robert0622

Copy link
Copy Markdown
Collaborator

LoopForge PR Review — head 24df82b8e58c9d1fdd5ee6728dd86b60bea5cac2

Verdict: ✅ Pass

  • Scope: 40 files, +1967/-226. Implements phased workspace-opening progress (build → clone → branch) with configurable per-phase timeouts (AGENTFLOW_WORKSPACE_<PHASE>_TIMEOUT), bounded+redacted command output surfaced to chat/run logs/problems via the new internal/workspaceprep package and procwatch.Event.Output, retains the workspace binding on failure so retry reuses the same run/checkout, and adds idempotent branch creation on retry (repository_checkout.go: skips createNewRepositoryBranch when the ref already exists or an agentflow-baseline marker is present). Addresses [Bug]: 工作区克隆仓库或构建沙箱镜像很慢时一直显示「打开中」,没有进度、超时和失败提示 #124's phase/timeout/error-surfacing asks; repo-mirror caching and "notify when Dockerfile changed, offer rebuild" from the issue's broader wish list are outside this PR's stated scope.
  • Verified in an isolated sandbox (no secrets, no production access), checked out this exact head via git worktree:
    • cd chatbot && npm ci && npx vitest run — 336/336 tests passed (52 files), including the new tests/workspace-opening.test.jsx (4/4).
    • Read-through of internal/workspaceprep/progress.go (timeout/error wrapping, context-key sink, regex-based secret redaction with rune-safe truncation) and internal/procwatch/procwatch.go (bounded tail by bytes/lines, oversized-line discard so a credential split across writes can't leak a fragment) — no correctness issues found; both ship dedicated unit tests (progress_test.go, procwatch_test.go, workspace_preparation_timeout_test.go, observer_test.go).
    • Could not run go build/go test locally — this sandbox has no Go toolchain and go.mod pins go 1.26.11, which isn't installable offline here. Relying on CI for Go-side evidence: platform-required, e2e-required, core, browser, first-run, validate (3.8/3.12) all green on this exact head.
  • No blocking issues found within the reviewed scope.

Keep generation diagnostics alongside workspace-opening progress, and retain procwatch error helpers with bounded failure output.
@AlexXPZhu
AlexXPZhu dismissed a stale review via 477c6cb November 13, 2026 12:43
@Robert0622

Copy link
Copy Markdown
Collaborator

LoopForge PR Review — head 477c6cbdc1240f4b56e7d09d805c19544ef210a7

Verdict: ❌ Fail

Blocking: please rebase/re-merge main and reconcile this PR's workspace.progress/workspace.cancel handling in runner_ws.go / control.go with #494's agent_tool.* conversation-gateway additions, then push so CI reruns on a clean head — @AlexXPZhu.

Merge main through eb4d4d5 to resolve the overlap with Tencent#494. Keep opening
progress and request-scoped cancellation alongside agent-tool forwarding,
serialized websocket writes, and refreshed write deadlines.

Cancel one-shot preparation and checkpoint requests without terminating
persistent conversation processes when their start RPC completes. Preserve
bounded cancellation delivery when the control writer is busy.

Guard optional progress recording during SOP handoff and bind the retry
fixture to its session's execution target. Cover concurrent tool forwarding,
CLI lifetime, checkpoint release, and cancellation backpressure.
@AlexXPZhu
AlexXPZhu dismissed a stale review via e2ab6ee November 13, 2026 16:52
cocofhu and others added 2 commits November 14, 2026 01:41
The merge with Tencent#494 limits remote cancellation and live opening progress to
checkpoint work and workspace-kind preparation. No test exercised a
run-preparation request (RunKind other than workspace), so dropping the
RunKind check on either side went unnoticed.

- httpapi: TestRunPreparationIsNotRequestScoped checks the hub neither
  forwards opening progress nor sends workspace.cancel for such a request.
- runner: TestWorkspaceCancelDoesNotAbortRunPreparation checks a stray
  workspace.cancel does not abort the preparation script or stream progress.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sanitize only matched the Basic and Bearer schemes, so an
"Authorization: token <pat>" header echoed in command output reached the
progress boundary unredacted; the receiving side's exact-token redaction
was the only safeguard. Match the token scheme as well and cover both it
and an http.extraHeader Basic header in the sanitizer test.
@Robert0622

Copy link
Copy Markdown
Collaborator

LoopForge PR Review — head f9ffef80aed2348b459f4946acdb07484e1242c2

Verdict: ✅ Pass

  • Re-review: head moved from 477c6cbdc1240f4b56e7d09d805c19544ef210a7 (previously Fail — real merge conflicts with main in internal/httpapi/runner_ws.go / internal/runner/control.go colliding with feat(chat): run native agent CLIs in conversation workspaces on bound execution machines #494's agent_tool.* plumbing) through e2ab6ee (merges main through eb4d4d5, reconciling workspace-progress/cancellation with the native CLI runner-control additions: keeps agent_tool.request/response forwarding, serializes websocket writes, and scopes one-shot preparation/checkpoint cancellation so persistent conversation processes aren't terminated by it), 3dadb38 (test pin) and f9ffef8 (adds the token scheme to the Authorization header redaction regex, with a test covering both Authorization: token <pat> and http.extraHeader=Authorization: Basic ...).
  • Confirmed the conflict is actually resolved: git merge-tree origin/main pr-149 is clean (no conflict markers), and GitHub reports mergeable: true / mergeable_state: clean on this exact head.
  • Required GitHub checks: 16/16 green on this head (platform-required, e2e-required, core, browser, validate 3.8/3.12, secrets, curl-install, first-run, tcmcp, 4×e2e, e2e-browser, opensource scan).
  • Ran locally in an isolated, credential-free sandbox (no Go toolchain available here, so relying on CI for Go-side evidence): cd chatbot && npm ci && npx vitest run — full suite passed 372/372 (56 files) on a clean rerun; one earlier run showed a single unrelated failure in tests/app-render.test.jsx (an image-preview case), which did not reproduce on retry and also doesn't reproduce on main's 368-test suite — looks like cross-test ordering flakiness, not something introduced by this PR's logic or its new workspace-opening.test.jsx.
  • Read through the e2ab6ee reconciliation diff directly (runner_ws.go's send/resolveWorkspaceProgress, control.go's readControls/conversationForward) — the merge correctly keeps both features: progress polling + bounded cancellation delivery for one-shot prepare/checkpoint requests, and agent_tool.* request/response forwarding for persistent conversation processes, without regressing either.
  • No blocking issues found within the reviewed scope.

Yanami-2K
Yanami-2K previously approved these changes Nov 14, 2026

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对照 #124 核对了需求:分阶段进度(准备/构建/克隆/分支)、各阶段可配置超时、失败写入运行日志和 runs_get.problems、重试复用同一 run 与已完成检出、Dockerfile 冻结版本提示、慢网络文档,这些都有对应实现;全量 mirror 缓存未做,PR 描述里已经说明。RecordWorkspaceOpening 在落库前对 message/output 做 token 脱敏和长度限制;RemoveIncompleteClone 只删除空的中断克隆,有文件、index、baseline 或 refs 时都会保留;workspacePreparationProblems 也处理了“opening 已就绪后清除旧失败”和“之后又有新失败”的去重。新增测试断言了脱敏、StartedAt 延续、事件压缩、阶段超时和 SSH 密钥约束,CI 全绿、mergeableState=clean,所以批准。

说明:本次 diff 里有约 20 个文件的 patch 被省略,没能逐行看到,包括 internal/procwatch/procwatch.go、internal/runner/daemon.go、internal/runner/control.go、internal/runner/workspace_preparation.go、internal/repo/runs.go、internal/workspaceprep/*.go,以及 internal/httpapi/workspace_opening_test.go。这些文件的判断依据是 PR 描述和对应的测试名,合并前建议维护者重点再看 runner 的 workspace.progress/workspace.cancel 协议部分。

Merge main through 777f16a while preserving workspace progress, scoped
cancellation, and retries that retain the current workspace and its files.
Keep upstream's sandbox-only opening API and legacy Git-config restoration.

Adapt retry guidance to the current environment-selection contract and
explain the scope of platform-managed Git deadlines. Update the Docker
deadline fixture for direct script execution and workspace mount inspection,
and verify that a failed empty workspace is reused and becomes ready on retry.
@AlexXPZhu
AlexXPZhu dismissed stale reviews from Yanami-2K and ghost via a9d9ef8 November 16, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: 工作区克隆仓库或构建沙箱镜像很慢时一直显示「打开中」,没有进度、超时和失败提示

4 participants