Skip to content

feat(website): WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) - #428

Open
williamloo-scale wants to merge 5 commits into
wloo/website-harness-gapsfrom
wloo/website-environment-sdk
Open

williamloo-scale wants to merge 5 commits into
wloo/website-harness-gapsfrom
wloo/website-environment-sdk

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #427.

agentenv_protocol.website: the easy path to a synthetic website that follows website/v1 (urn:agentenv:website/v1).

@environment_card(name="notes")
@website(domains=["notes.test"], static="web/dist", health="/healthz", data="data/seed.json")
class Notes(WebsiteEnvironment):
    @api.get("/api/notes")
    def list_notes(self, ctx: ApiContext) -> list:
        return self.store.find("notes", owner=ctx.user)
  • @api.get/post/put/patch/delete(path, name=, roles=, status=): path, query and body are bound from type hints, errors are exceptions mapped to statuses, and an OpenAPI 3.1 document is generated.
  • ApiContext: the signed-in user (else the acting user, so the agent starts signed in), the caller's role, and the public URL from X-Forwarded-Host/-Proto.
  • Roles: the gateway forwards whatever AgentEnv-Role a browser sends, so a site believes a non-default role only when the request also carries AgentEnv-Role-Key equal to the site's WEBSITE_ROLE_KEY (compared in constant time). Every other caller, and every caller of a site with no key set, is default, so roles= routes are closed until a key is set.
  • Serves the built pages with an index.html fallback, the health path with @ready_check, the data plane and a card declaring website/v1 with its routes and openapi/session operations. /agentenv… routes are matched before the site's own.
  • self.now() reads the universe clock (clock/v1 sync_time); every response carries X-Universe-Now; clock_patch=True patches the page's Date. sync_time also takes main's optional timezone (feat(sync_env_clock): an optional timezone reads virtual_time in the zone and tells every synced server (FD-5003) #442), which the site reads as self.timezone; now() stays UTC.
  • Collections store: load, export and reset with no data hooks written. A site's data is read and written only through the data plane (data/get, data/add, data/reset). The SDK serves no /go and no website/v1 state op.
  • website_state_verifier, which feat(website): a website env is one server image, checked at its health path (FD-4683) #390 merged and which read a site's /go?sid=, is removed. Grading a website from data/get against the task's initial state moves to agentenvhub-sdk's cua_gym_verifier, a separate PR in scaleapi2.
  • python -m agentenv_protocol.website.conformance URL checks any server over HTTP: card, any host name, health, universe time, data plane, clock, forwarded headers, cookies.

Known gaps

Each is fixed later in the stack, as noted.

Test

Unit suite and protocol tests: 12,659 passed, lint clean.
packages/agentenv-protocol/tests: 946 passed; the website tests (59, with the Notes ones) pass after the role-key fix. Deployed locally behind the gateway, a spoofed AgentEnv-Role: admin now gets 403 on both /website/ and /svc/ (200 before).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge, with all three numbered findings addressed and no new blocking issue found.

Summary

This PR adds WebsiteEnvironment, typed HTTP routes, sessions, a collections store, shared readiness checks, and an HTTP conformance checker. It also removes website_state_verifier.

  • WebsiteEnvironment serves pages and typed routes on website/v1.
  • A conformance command checks whether a server follows website/v1.
  • Deployments and env records read readiness from each website's card.
  • The gateway blocks website browsing routes advertised for agentenv.
  • Env callers can tell apart website and MCP children with the same name.
  • The website state verifier is removed from task steps.

Reviews (14) · Last reviewed commit: "fix(website): leave a partial HTML answe..." · Reviewed by Greptile

williamloo-scale added a commit that referenced this pull request Nov 3, 2026
Brings in #428's review fixes beside this branch's: the clock's sync
reads under the refresh lock and keeps the answered rate; conformance
checks X-Universe-Now last, on every answer, at the recommended level,
with the reserved-path probes counted; the handwritten test server
stamps its 404s and 405s like every other answer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tract, and a conformance check tests any server (FD-4683)

This PR's change, rebased as one commit onto #427.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/agent_env/env/env.py Resolved
@williamloo-scale
williamloo-scale force-pushed the wloo/website-harness-gaps branch from 7fc14cd to b18049d Compare November 3, 2026 21:50
@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 3579c2a to 76145d2 Compare November 3, 2026 21:50
Comment thread src/agent_env/env/gateway/gateway.py Resolved
williamloo-scale and others added 2 commits November 3, 2026 14:20
… keep older sites and shared names working (FD-4683)

- The gateway answers 503 on a site's route while its card can't be read, rather than forwarding with nothing reserved.
- A plugin's MultiEnv record looks each child up by its kind, so a website and an MCP server may share a name.
- The state verifier reads a site its definition names but its card doesn't list at /svc/<name>/go, deprecated.
- The clock script is put in streamed HTML holding only what comes before <head>.
- Conformance fails a website/v1 ready that answers an HTTP error, whatever its body says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ver HTTP client the test client uses (FD-4683)

After a sign-out the client's jar keeps the site's signed-out cookie under testserver.local, and setting the
forged one adds a second cookie of the same name. httpx2, which Starlette's TestClient uses when it is
installed (as on CodeBuild), sends the forged one first, so the site reads the valid signed-out cookie and
the test fails; httpx sends them the other way round. Clearing the jar first sends the forged cookie alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator

This is a huge red flag: it leaks CUA-Gym into agentenv-framework. /go?sid= is CUA-Gym's verifier endpoint, and this PR builds it into the SDK and the website/v1 contract rather than keeping it outside the framework:

  • The SDK: agentenv_protocol.website defines and exports STATE_PATH = "/go" (environment.py:51, __init__.py:15). Every store-backed SDK site serves its state there (environment.py:173) and advertises it as website/v1's params.state.path (environment.py:198).
  • Conformance: the checker reads state at params.state.path, else at /go (conformance.py:200-208), so it judges any server by CUA-Gym's convention.
  • Verifier and gateway: website_state_verifier falls back to params.state.path, else /go?sid= (website_state_verifier.py:37, :279-283). The gateway's guard is built around params.state.path (gateway.py:1556). The /go fallback predates this PR, but this PR extends it.

website/v1's state op should be the only state contract: an SDK site answers its state only through it. Please drop /go and params.state.path from the SDK, the conformance check and website/v1. If CUA-Gym sites still need to be read at /go, that compatibility belongs in a CUA-Gym adapter outside the framework, not in agentenv_protocol or the core verifier.

@earakely-scale

Copy link
Copy Markdown
Collaborator

Following up on my /go comment: please remove CUA-Gym's verifier model from the framework altogether. That model is a site serving {initial_state, current_state, state_diff} per sid, and a verifier reading it. This goes further than my earlier note, so drop the website/v1 state op too, since it's the same model under another name. agentenv already has the primitives for this:

What the verifier needs agentenv primitive
current state the data plane's data/get, which SDK sites already serve from the same store (_store_get)
initial state the universe the task loaded with data/add, or a snapshot_env taken before the agent starts
diff computed once, in the verifier, from those two, the same for every env instead of each site writing its own (that also removes the store-id diff problem)
sid not needed: each deploy gets its own site, and the SDK already ignores it
keeping state from the agent the data plane is framework-only and already refused on the browsing route

Concretely:

  • SDK: remove the /go route, STATE_PATH, params.state.path, the state op, and the store's site-side state and diff (Collections.state() / diff()). A site exposes its data through data/get only.
  • Verifier: grade by reading data/get against the initial state, keeping the check model (path, op, value, reduce, weight). It then works for any env, MCP servers included, not only websites.
  • Conformance and gateway: check data/get rather than /go or state. Drop the params.state.path reservation.
  • Main: website_state_verifier as feat(website): a website env is one server image, checked at its health path (FD-4683) #390 merged it reads /go?sid=. Replace it with the above rather than extending it. CUA-Gym images that serve only /go get an adapter outside the framework.

@williamloo-scale

williamloo-scale commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Agreed, and done in this stack rather than deferred. /go?sid= and the per-site {initial_state, current_state, state_diff} model were CUA-Gym's, and they're gone from the framework:

Each PR removes what it had added, so the stack never introduces any of this. Grading a website now belongs outside the framework: cua_gym_verifier in agentenvhub-sdk, in a separate scaleapi2 PR. It reads data/get, diffs it against the task's initial state (the universe it loaded or a snapshot_env), and keeps the check model: path, op, value, reduce and weight.

@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 1ffec4c to 05da818 Compare November 4, 2026 03:37
…op /go, the state op and website_state_verifier (FD-4683)

- The SDK serves no `/go`, no `params.state.path` and no `website/v1` `state` op; `STATE_PATH`, `WebsiteState`,
  `website_state`, `website_sites` and `website/state.py` are gone, and the default store keeps no initial state or diff.
- Conformance runs the data plane's round trip on every server, with no skip for a state that shows changes.
- The gateway reserves the card's `url` and every advertised endpoint, no longer a state path.
- `website_state_verifier` is removed from the framework; grading against a site's state belongs outside it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 05da818 to 33ad3b8 Compare November 4, 2026 03:56
… items in OpenAPI, and find the clock's sync_time by name (FD-4683)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale

Copy link
Copy Markdown
Contributor Author

@greptile-apps all three remaining review comments are fixed in d4efdbb, each with a test that fails without its fix:

  • Partial HTML downloads change (environment.py): _Universe no longer injects the clock script into a partial answer. A 206, or any answer with Content-Range, passes through unchanged, so its bytes still match the promised range.
  • Query lists lose item types (routes.py): a list[T] query parameter is documented as {"type": "array", "items": <T's schema>}, and it still binds as repeated values.
  • Working clocks can fail (conformance.py): _clock() looks up sync_time by name instead of taking the first operation the card lists.

The full unit suite passes on this commit (12,908) and lint is clean. All three threads are replied to and resolved.

@greptile-apps

greptile-apps Bot commented Nov 4, 2026

Copy link
Copy Markdown

Confirmed: all three findings are fixed in d4efdbbf and have regression coverage:

  • Partial HTML downloads: _Universe skips clock-script injection for status 206 and responses with Content-Range; the test verifies a range response remains unchanged.
  • Query-list item types: OpenAPI now includes the item schema for list query parameters, while binding still uses repeated query values; the test checks list[int] documentation and binding.
  • Clock operation selection: the checker finds sync_time by name; the test passes when another clock operation is listed first.

These comments can remain resolved. I verified the code and tests in the commit; I did not independently run the suite.

williamloo-scale added a commit that referenced this pull request Nov 4, 2026
…ses the conformance check (FD-4683)

This PR's change, rebased as one commit onto #428.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants