Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions packages/agentenv-protocol/examples/website/replay/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
ARG REPLAY=packages/agentenv-protocol/examples/website/replay

FROM mirror.gcr.io/library/python:3.12-slim@sha256:423ed6ab25b1921a477529254bfeeabf5855151dc2c3141699a1bfc852199fbf
ARG REPLAY
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
COPY packages/agentenv-protocol /tmp/agentenv-protocol
RUN pip install --no-cache-dir /tmp/agentenv-protocol "uvicorn>=0.30" && rm -rf /tmp/agentenv-protocol
COPY ${REPLAY}/site.py ${REPLAY}/capture.warc.gz /app/
ENV PORT=8000
EXPOSE 8000
RUN useradd --create-home --uid 10001 appuser
USER appuser
CMD ["python", "site.py"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
**/node_modules
**/dist
**/__pycache__
**/.pytest_cache
**/.venv
.git
78 changes: 78 additions & 0 deletions packages/agentenv-protocol/examples/website/replay/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Widget Docs: a captured site, replayed

Widget Docs is a small docs site captured to a WARC file and served by
`agentenv_protocol.website.replay.ReplayWebsite`. The captured pages are served as they were captured, for any host,
with only their same-site links rewritten to stay on the replay. The newsletter sign-up, which a capture can't answer, is written as `@api` routes on top.

| File | What it is |
| --- | --- |
| `source/` | The site that was captured: a home page, two guide pages, a stylesheet and a script. |
| `capture.warc.gz` | The capture of `source/` at `--depth 1`: `/`, `/guide/install.html`, `/guide/usage.html?utm_source=home`, `/style.css` and `/app.js`. `/guide/advanced.html` is two links from home, so it isn't in the capture. |
| `Dockerfile`, `env.toml` | The server image, built from the repo root as the Notes example's is, and the website env as a bundle entry. |
| `site.py` | The server: `WidgetDocs(ReplayWebsite)` with `capture = "capture.warc.gz"`, plus `POST /api/subscribe` and `GET /api/subscriptions`. |

## Capture a site

```bash
python -m agentenv_protocol.website.replay capture https://docs.example.com/ --out capture.warc.gz --depth 2
```

The tool needs only this package and `httpx`. It fetches each start URL. It then fetches the same-site pages those URLs
link to, up to `--depth` links away, and the assets (stylesheets, scripts, images) each captured page loads, with the
stylesheets, images and fonts a captured stylesheet's `@import` and `url()` name. Every
answer goes into a WARC 1.1 file, gzipped one record at a time, which pywb and warcio also read. Notes:

- Only GETs are captured.
- Redirects are kept as they were answered, and their targets are captured too.
- Bodies are stored decoded, so the stored headers leave out `Content-Encoding` and `Transfer-Encoding`.
- A host is the same site as a start URL when it is that URL's host (without `www.`) or a subdomain of it.
- Request records name only the method, the target and `Host`, so no credential the capturing client sent ends up in
the file.
- `--max-requests` (500 by default) caps a capture, and `--no-assets` leaves out what pages load.

To capture `source/` again:

```bash
python -m http.server -d source 8765 &
python -m agentenv_protocol.website.replay capture http://127.0.0.1:8765/ --out capture.warc.gz --depth 1
```

## How a request is matched

A request is matched by its method (HEAD as GET), its path and its query, never by its host or headers. The query is
normalised before matching:

- Its parameters are sorted.
- Blank parameters are kept.
- The `utm_*`, `fbclid`, `gclid`, `msclkid` and `_` parameters are dropped. Set `ignore_query` on the class to change
this list.

Each path segment is percent-decoded and then encoded again, so `%7E` and `~` match. A request the capture doesn't
hold gets a 404. If a capture holds one request twice, the first answer is served.

When the capture's answers are served:

- `Set-Cookie` loses its `Domain` attribute, so the cookie works on the replay's own host.
- A `Location` that points at a captured host is made relative.
- In HTML pages (`href`, `src`, `srcset`, `action`, `formaction`, `poster`, `style` and `<style>`) and stylesheets
(`url()`, `@import`), a URL on a captured host is made root-relative, and a root-relative URL is led by the prefix
the replay is served under (`X-Forwarded-Prefix`, such as `/website/<name>`). Script bodies aren't changed.
- `Strict-Transport-Security` and `Alt-Svc` are dropped.

## Synthetic routes on top

`@api` routes win over the capture. Here `POST /api/subscribe` keeps sign-ups in a list of the site's own. Its
`@get_data` exports them through `data/get`, which is how a run's sign-ups are read; `data/reset` empties them through
`reset_site_data`, and a `data/add` data part's `subscriptions` are added through `add_site_data`. The capture itself
isn't task state. A `data/add` file part named `*.warc` or `*.warc.gz` (or typed `application/warc`)
replaces the declared capture until the next `data/reset`. This lets a large capture be loaded at deploy time instead
of being built into the image.

## Run and test it

```bash
PYTHONPATH=<repo>/packages/agentenv-protocol/src python site.py # serves on PORT, 8000 by default
python -m agentenv_protocol.website.replay serve capture.warc.gz --domain docs.widgets.test # a bare replay
python -m pytest packages/agentenv-protocol/tests/test_website_replay.py
docker build -f packages/agentenv-protocol/examples/website/replay/Dockerfile -t widget-docs . # from the repo root
```
Binary file not shown.
3 changes: 3 additions & 0 deletions packages/agentenv-protocol/examples/website/replay/env.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
type = "website"
environment_name = "widget-docs"
backend_image = { dockerfile = "Dockerfile" }
70 changes: 70 additions & 0 deletions packages/agentenv-protocol/examples/website/replay/site.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
"""Widget Docs, replayed: the pages of a captured docs site, served as they were captured for any host, with the
newsletter sign-up the capture can't answer written as ``@api`` routes on top. The sign-ups are the site's state, kept
behind one lock since plain ``def`` handlers run in worker threads: the data plane's ``data/get`` exports them and
``data/reset`` empties them."""
from __future__ import annotations

import copy
import threading

from pydantic import BaseModel

from agentenv_protocol import DataPart, environment_card, get_data
from agentenv_protocol.website import BadRequest, Conflict, api, website
from agentenv_protocol.website.replay import ReplayWebsite

SUBSCRIPTIONS = "subscriptions"


class Subscription(BaseModel):
email: str


@environment_card(name="widget-docs")
@website(domains=["docs.widgets.test"])
class WidgetDocs(ReplayWebsite):
capture = "capture.warc.gz"

def __init__(self) -> None:
super().__init__()
self._lock = threading.Lock()
self.subscriptions: list[dict] = []

def add_site_data(self, parts: list) -> None:
"""Each data part's ``subscriptions``, added to those there are."""
with self._lock:
for part in parts:
if not isinstance(part, DataPart):
raise ValueError(f"Widget Docs loads its sign-ups from a data part, not a {type(part).__name__}")
self.subscriptions += copy.deepcopy(part.data.get(SUBSCRIPTIONS, []))

def reset_site_data(self) -> None:
with self._lock:
self.subscriptions = []

@get_data
def export(self) -> list:
with self._lock:
return [DataPart(data={SUBSCRIPTIONS: copy.deepcopy(self.subscriptions)})]

@api.post("/api/subscribe", status=201)
def subscribe(self, body: Subscription) -> dict:
"""Sign an email address up for the newsletter."""
if "@" not in body.email:
raise BadRequest("that is not an email address")
with self._lock:
if any(s["email"] == body.email for s in self.subscriptions):
raise Conflict(f"{body.email} is subscribed already")
subscribed = {"id": len(self.subscriptions) + 1, "email": body.email, "at": self.now().isoformat()}
self.subscriptions.append(subscribed)
return copy.deepcopy(subscribed)

@api.get("/api/subscriptions")
def list_subscriptions(self) -> list[dict]:
"""Everyone signed up."""
with self._lock:
return copy.deepcopy(self.subscriptions)


if __name__ == "__main__":
WidgetDocs().serve()
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
document.getElementById("subscribe").addEventListener("submit", async (event) => {
event.preventDefault();
const email = new FormData(event.target).get("email");
const answer = await fetch("api/subscribe", {
method: "POST", headers: {"Content-Type": "application/json"}, body: JSON.stringify({email}),
});
document.getElementById("subscribed").textContent = answer.ok ? `Subscribed ${email}` : "Try again";
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
<!doctype html>
<html>
<head><meta charset="utf-8"><title>Advanced</title></head>
<body><h1>Advanced</h1><p>Two links from the home page, so a capture at --depth 1 leaves it out.</p></body>
</html>
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<!doctype html>
<html>
<head><meta charset="utf-8"><title>Install</title><link rel="stylesheet" href="../style.css"></head>
<body>
<h1>Install</h1>
<pre>pip install widgets</pre>
<p><a href="usage.html">Next: usage</a> &middot; <a href="../">Home</a></p>
</body>
</html>
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<!doctype html>
<html>
<head><meta charset="utf-8"><title>Usage</title><link rel="stylesheet" href="../style.css"></head>
<body>
<h1>Usage</h1>
<pre>import widgets; widgets.spin()</pre>
<p><a href="advanced.html">Advanced</a> &middot; <a href="../">Home</a></p>
</body>
</html>
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Widget Docs</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<h1>Widget Docs</h1>
<p>Everything about widgets.</p>
<ul>
<li><a href="guide/install.html">Install</a></li>
<li><a href="guide/usage.html?utm_source=home">Usage</a></li>
<li><a href="https://elsewhere.example/">Another site</a></li>
</ul>
<form id="subscribe">
<input name="email" type="email" placeholder="you@example.com">
<button>Subscribe</button>
</form>
<p id="subscribed"></p>
<script src="app.js"></script>
</body>
</html>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
body { font-family: sans-serif; max-width: 40rem; margin: 2rem auto; }
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
``env_readiness`` of every site of an env. What a task left a site with is its data plane's ``data/get``. A site's pages
can call its API with ``client/site-client.js``, a dependency-free ES module (with its ``.d.ts``) to copy into them:
``api()``, ``universeNow()`` from ``X-Universe-Now``, ``followUniverseClock()``, ``session()``, ``signIn()``
and ``signOut()``."""
and ``signOut()``. A live site, ``@website(domains=[...], live=True, allow_hosts=[...])``, serves none of its pages and
signs a browser in with ``@session``; ``agentenv_protocol.website.replay`` captures a real site and serves the capture
as ``ReplayWebsite``."""

from __future__ import annotations

Expand All @@ -17,6 +19,8 @@
WebsiteEnvironment,
host_problem,
ready_check,
session,
session_state,
website,
)
from .errors import ApiError, BadRequest, Conflict, Forbidden, NotFound, Unauthorized
Expand All @@ -35,7 +39,8 @@
from .session import SESSION_COOKIE, SESSION_SECRET_ENV, SessionSigner

__all__ = [
"WebsiteEnvironment", "WebsiteDeclaration", "website", "api", "ApiContext", "ApiRoute", "ready_check", "host_problem",
"WebsiteEnvironment", "WebsiteDeclaration", "website", "api", "ApiContext", "ApiRoute", "ready_check", "session",
"session_state", "host_problem",
"ApiError", "BadRequest", "Unauthorized", "Forbidden", "NotFound", "Conflict",
"UniverseClock", "ClockUnavailable", "SessionSigner", "reserved_path",
"WEBSITE_EXTENSION_URI", "CLOCK_EXTENSION_URI", "UNIVERSE_NOW_HEADER", "DEFAULT_HEALTH_PATH", "SESSION_PATH",
Expand Down
Loading
Loading