Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions performance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
against file watching.
- [agents](agents/README.md): what the CLIs cost the AI agent that reads their answers — output size and meeting workflows.
- [tasks](tasks/README.md): tasks found in messages — duplicate detection.
- [command-server](command-server/README.md): `commands --serve` for WireCat Desktop — several requests at once
through one process.
- [startup](startup/README.md): install size and cold start of the five CLIs, and which packages a simple
command loads.
- [projects](projects/README.md): gathering cited evidence about a project — what it finds against a frozen
Expand Down
6 changes: 6 additions & 0 deletions performance/command-server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Command server

Measurements of `<cli> commands --serve`, the long-running process WireCat Desktop sends its pages' commands to.

- [concurrent](concurrent/README.md): 7 reads through one server, one at a time and at once, against a process for
each: about 0.1 s against 0.7–1 s, and running them at once saves another 6–54 % (2026-11-10).
70 changes: 70 additions & 0 deletions performance/command-server/concurrent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Several requests at once through one command server

Measured 2026-11-10 on cli-core, with [`measure.mjs`](measure.mjs) beside this page. The raw output is in
[`results.txt`](results.txt).

## Why we ran it

WireCat Desktop answers its pages through `memo commands --serve`. Before this change, one server answered one request
at a time. Desktop keeps a pool of 2 servers. The Memory page sends 7 reads together, so 5 of them start a fresh `memo`
process each, which takes about 0.4 to 0.9 s and about 165 MB. The owner asked why one server could not take
parallel calls and leave the locking to SQLite.

The questions:

1. How long do 7 reads take through one server when they run one at a time, and when they run at once?
2. How does that compare with a process for each read, which is what Desktop pays for the reads that miss the pool?

The answers fed [cli-core #2812](https://github.com/WireCatLabs/cli-core/pull/2812): up to 8 requests at once in one
server, and the change Desktop needs to send them.

## Setup

- **Machine.** AMD Ryzen AI 9 HX 470 (24 threads), 30 GB RAM, Linux, Node 24.19.0. Other agents ran tests at the
same time: the load average was about 8.5 (written in `results.txt`). Nothing was pinned to CPUs.
- **Code.** cli-core on the branch of #2812 (`3319e0ddc`), the test CLI `src/testing/command-server-cli.ts` built with
esbuild by the script. It is a tg-like CLI on cli-core's `run()`.
- **State.** A throwaway folder for `HOME`, the config, state and cache folders and `MESSAGING_STORE`, with
`DBUS_SESSION_BUS_ADDRESS=disabled:`. The store holds 30 saved searches and nothing else. No real store, account or
message was opened.
- **Commands.** `work searches list --json` (settings, the store, a list of 30 rows) and `work config show --json`
(settings only). Each round is timed from the first request sent to the last answer read. The server gets one
request first, so its modules are loaded, as for a page's call after the server started.

```sh
# from a cli-core checkout with #2812 (the script is also at bench/command-server/measure.mjs there)
node bench/command-server/measure.mjs 7 5 # 7 reads, 5 rounds, searches list
node bench/command-server/measure.mjs 7 5 work config show --json # the same for config show
node bench/command-server/measure.mjs 16 3 # 16 reads, past the cap of 8
```

## Results

Medians, in ms. The ranges are the three runs made that day (two before `results.txt`, one in it).

| 7 reads | `searches list` | `config show` |
|---|---|---|
| one server, one at a time | 95–157 | 91–164 |
| one server, all at once | 76–148 | 70–97 |
| a process each, all at once | 671–1040 | 676–830 |
| a process each, one at a time | 2985–5271 | 3271–3992 |

16 reads of `searches list`: 338–407 ms one at a time through one server, 241–274 ms all at once, 1482 ms with a
process each, all at once.

## What it showed

- **Most of the gain is not starting a process.** 7 reads through one server take about 0.1 s in any order. 7
processes started together take 0.7 to 1 s.
- **Running them at once saves another 6 to 54 %.** The work is mostly synchronous JavaScript and SQLite on one
thread, so requests overlap only where a run waits: on files, settings and imports.
- So Desktop can get most of the gain by sending a page's reads to one server instead of starting processes. The
answers come back by `id`, in the order they end.

## What was not measured

- A real memo store, and the Memory page's actual 7 commands. The store here is tiny.
- Memory use of one server with 8 requests in flight.
- A server while another process holds the store's write lock. The driver is synchronous, so such a wait stops the
whole server for up to the busy timeout (5 s), not only the request that waits.
- Other machines, and runs pinned to fewer CPUs as CI is.
104 changes: 104 additions & 0 deletions performance/command-server/concurrent/measure.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
// N reads through one `commands --serve`, one at a time and all at once, against a process each.
// Run from the repository root: node bench/command-server/measure.mjs [n=7] [rounds=5] [argv...]
// It builds the test CLI (src/testing/command-server-cli.ts) and uses a throwaway store and HOME.
import { execFileSync, spawn } from "node:child_process"
import { mkdtempSync } from "node:fs"
import { tmpdir } from "node:os"
import { join } from "node:path"
import { createInterface } from "node:readline"
import { build } from "esbuild"

const [nText = "7", roundsText = "5", ...given] = process.argv.slice(2)
const n = Number(nText)
const rounds = Number(roundsText)
const argv = given.length > 0 ? given : ["work", "searches", "list", "--json"]

const root = mkdtempSync(join(tmpdir(), "command-server-bench-"))
await build({
entryPoints: ["src/testing/command-server-cli.ts"],
outdir: join(root, "cli"),
bundle: true,
splitting: true,
platform: "node",
format: "esm",
logLevel: "error",
plugins: [
{
name: "packages-in-place",
setup(builder) {
builder.onResolve({ filter: /^[^./]/ }, ({ path }) =>
/^(node|bun):/.test(path) ? { external: true } : { path: import.meta.resolve(path), external: true },
)
},
},
],
})
const cli = join(root, "cli", "command-server-cli.js")
const env = {
...process.env,
HOME: join(root, "home"),
MESSAGING_STORE: join(root, "store.db"),
CHAT_CONFIG_DIR: join(root, "config"),
CHAT_STATE_DIR: join(root, "state"),
CHAT_CACHE_DIR: join(root, "cache"),
CHAT_PROFILE: "",
DBUS_SESSION_BUS_ADDRESS: "disabled:",
WIRECAT_CALLER: "desktop",
}
for (let at = 0; at < 30; at++)
execFileSync(process.execPath, [cli, "work", "searches", "create", `saved-${at}`, `word${at} other`], { env })

const spawned = () =>
new Promise((resolve) => {
const child = spawn(process.execPath, [cli, ...argv], { env, stdio: ["ignore", "pipe", "pipe"] })
child.stdout.resume()
child.stderr.resume()
child.on("close", resolve)
})

const server = async () => {
const child = spawn(process.execPath, [cli, "commands", "--serve"], { env, stdio: ["pipe", "pipe", "inherit"] })
const waiting = new Map()
const ready = new Promise((resolve) => {
createInterface({ input: child.stdout }).on("line", (line) => {
const answer = JSON.parse(line)
if (answer.ready) return resolve(answer)
if (answer.code !== 0) process.stderr.write(`failed: ${answer.stderr}\n`)
waiting.get(answer.id)?.(answer)
waiting.delete(answer.id)
})
})
process.stdout.write(`${JSON.stringify(await ready)}\n`)
let id = 0
const ask = () =>
new Promise((resolve) => {
id += 1
waiting.set(id, resolve)
child.stdin.write(`${JSON.stringify({ id, argv })}\n`)
})
return { ask, stop: () => child.stdin.end() }
}

const time = async (label, body) => {
const took = []
for (let round = 0; round < rounds; round++) {
const started = performance.now()
await body()
took.push(performance.now() - started)
}
took.sort((a, b) => a - b)
process.stdout.write(`${label}: median ${took[Math.floor(took.length / 2)].toFixed(0)} ms over ${rounds} rounds\n`)
}

const one = await server()
// The first request loads the program's modules, as a page's first call after the server starts.
await one.ask()
await time(`${n} reads, one server, one at a time`, async () => {
for (let at = 0; at < n; at++) await one.ask()
})
await time(`${n} reads, one server, all at once`, () => Promise.all(Array.from({ length: n }, one.ask)))
one.stop()
await time(`${n} reads, a process each, all at once`, () => Promise.all(Array.from({ length: n }, spawned)))
await time(`${n} reads, a process each, one at a time`, async () => {
for (let at = 0; at < n; at++) await spawned()
})
19 changes: 19 additions & 0 deletions performance/command-server/concurrent/results.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
cli-core 3319e0ddc, 2026-11-09T23:35:42Z, load 8.84 8.59 8.52
$ node bench/command-server/measure.mjs 7 5
{"ready":true,"protocol":1,"version":"1.0.0","concurrent":8}
7 reads, one server, one at a time: median 147 ms over 5 rounds
7 reads, one server, all at once: median 110 ms over 5 rounds
7 reads, a process each, all at once: median 804 ms over 5 rounds
7 reads, a process each, one at a time: median 3186 ms over 5 rounds
$ node bench/command-server/measure.mjs 7 5 work config show --json
{"ready":true,"protocol":1,"version":"1.0.0","concurrent":8}
7 reads, one server, one at a time: median 91 ms over 5 rounds
7 reads, one server, all at once: median 76 ms over 5 rounds
7 reads, a process each, all at once: median 676 ms over 5 rounds
7 reads, a process each, one at a time: median 3519 ms over 5 rounds
$ node bench/command-server/measure.mjs 16 3
{"ready":true,"protocol":1,"version":"1.0.0","concurrent":8}
16 reads, one server, one at a time: median 338 ms over 3 rounds
16 reads, one server, all at once: median 274 ms over 3 rounds
16 reads, a process each, all at once: median 1482 ms over 3 rounds
16 reads, a process each, one at a time: median 11119 ms over 3 rounds
Loading