[earakely-scale]

Summary

M8.21 (FD-4969), the first step toward Online-Mind2Web. A browser running alone loads any public site reliably, and each connection it makes is logged. Design note: Open-web hosts in the site proxy (M8.21, FD-4969); the note is settled once you've reviewed it.

Revised in review: the proxies refuse nothing, as upstream Harbor's public network mode refuses nothing. The sandbox's network policy decides where a browser alone can go, as it did before this PR. The internal-address check, the own_loopback setting and the probe's override for it are gone (the note's decisions 2, 3 and 7, withdrawn).

Before this PR: a browser alone with no site routing goes out through the headless base's stand-in proxy on 3128, which already forwarded every host, but:

• It dropped connections under load: it relayed with select(), which refuses a socket numbered past 1023, so a busy proxy dropped every connection it opened after that. CodeBuild's unit run caught it.
• It dropped idle connections at 60 s; the design asks for at least 120 s.
• It logged nothing, and neither did the shared site proxy's direct tunnel for a live site's hosts (feat(website): live website envs, real sites a multi env's computers reach at their own names (FD-4945) #579, on 3129).

What changes

The stand-in (ubuntu-24.04-headless/site-proxy.py)

• Relays with a selector (epoll on Linux), not select().
• Waits: 30 s for the lookup and every connection attempt together; an idle connection is kept 300 s, and a write to a peer that stopped reading ends then too.
• Answers a failure itself: a name that doesn't resolve, or a host that doesn't answer, is a 502 with X-Agentenv-Site-Proxy: failed.

Both proxies log the same JSON lines

• connection, when one ends: host, port, address, bytes each way, start, duration.
• failed, for one that couldn't open: method, host, port, reason (doesn't resolve or no answer) and the error.
• Every line is JSON: the stand-in's startup line, and the shared proxy's refusals (refused), unexpected connection errors (error) and restarts (restarted), which were plain text.
• The stand-in's log is /var/log/agentenv/site-proxy.log; the shared proxy's is /etc/agentenv/sites/site-proxy.log.
• They can't drift: the two proxies can't share code (the stand-in is baked into the base image and starts before the deploy installs site_proxy.py), so one test sends the same success and failures through both and checks the records match.

Unchanged: HTTPS is still tunnelled with CONNECT and never decrypted; plain HTTP is forwarded. A synthetic site still wins its names.

Docs: the headless base in computers/README.md, and the website env entry in AGENTS.md.

Test plan

• Unit:

  • test_computer_stand_in_proxy.py (the stand-in, one fake-network fixture for every case):
    • a public host tunnelled and forwarded, each connection logged;
    • nothing refused: the computer's own loopback and an internal address are reached alike;
    • a name that doesn't resolve answered with a logged 502;
    • a name's addresses tried in turn: the next one after a refusal, and all of them within one deadline when the first hangs;
    • a connection whose sockets are numbered past 1023 (it fails on the old select() relay);
    • a tunnel whose server stopped reading, ended within the idle timeout instead of waiting for ever.
  • test_computer_site_proxy.py:
    • a live site's host tunnelled and forwarded with each connection logged; one that doesn't answer in time is a logged 502;
    • refusals of hosts that are no site are JSON lines;
    • both proxies log a connection, a name that doesn't resolve and a host that doesn't answer in the same records.
  • Unit tiers: tst/unit/env, cli and the protocol tests pass here, apart from the Mac-only Windows disk-recipe test, which fails on main on this Mac too.

• Integration (CI green on 80601db, 32 checks, the remote CodeBuild tier included): test_alone_on_the_open_web_it_loads_a_public_site_through_its_proxy_and_an_agents_tab_and_logs_each_connection, a browser computer with no site routing:

  • https://github.com/ loads through the stand-in and in a CDP agent's own tab;
  • a name that doesn't resolve is a 502;
  • the log holds the github.com connection and the failure.
  • The browser-computer tests that serve their pages on the computer's own loopback pass with no setting again.
  • The site-routing test reads the shared proxy's refusals as JSON lines.

• VM e2e (c6i.4xlarge, amd64, local Docker), on d9e6e0a, before the log-format commit 80601db (its integration change, the routing test reading JSON refusals, runs in CI): 6 tests passed in 41 min (most of it waiting on archive.ubuntu.com, which was degraded):

  • the open-web test above;
  • both site-routing tests, feat(website): live website envs, real sites a multi env's computers reach at their own names (FD-4945) #579's live-host one among them;
  • three tests that serve their pages on the computer's own loopback, now with no setting: the agent's page tools, cookies and storage, and the data plane.

• VM chaos, one browser computer alone, passed:

  Phase	Result
  The stand-in stopped	a CDP agent's navigation gets net::ERR_PROXY_CONNECTION_FAILED: no fallback to a direct connection
  Started again	github.com loads
  Killed with kill -9	supervisord starts it again; github.com loads, and a name that doesn't resolve is still a 502 at once
  A host that never answers (10.255.255.1:81)	a 502 after 30 s, the connect deadline, never a hang
  localtest.me, a public name for 127.0.0.1	reaches a server on the computer's own loopback: nothing is refused
  20 fetches of github.com at once	all 200
  A tunnel idle for 90 s, past the old 60 s	still carries a TLS request: 200 Connection established, then 200 OK
  The log	22 github.com connections, localtest.me at 127.0.0.1, and both failures with their reasons

  The VM is torn down: instance terminated, security group and key pair deleted.

Not in this PR

• chromium-open-web itself: headful, network allow-all, alone (M8.24).
• The full request log of the traffic the proxy decrypts (M8.20, for WebArena-Verified).
• One proxy instead of two: replacing the stand-in with the shared proxy means holding the browser's launch until the deploy installs it.
• An immediate error for a host the sandbox's allowlist drops (M8.24).

🤖 Generated with Claude Code

[earakely-scale]

@greptileai review