perf(memo): people list with each person's kind kept in the store

performance/README.md

@@ -27,6 +27,6 @@
   cost without the network, MCP.
 - [ci](ci/README.md): how long the pull-request checks take, and where the time goes.
 - [crm](crm/README.md): CRM data across b24 and memo — relationships, deal parts, money and group locks, end to end.
-- [memo](memo/README.md): memo's own lists — paging past 500 rows, and what a brief answer saves.
+- [memo](memo/README.md): memo's own lists — paging past 500 rows, what a brief answer saves, and `people list` with kinds kept in the store.
 - [funnel](funnel/README.md): wirecat-funnel, the tsnet helper for Tailscale Funnel: whether Tailscale's
   log upload stays off, and Funnel end to end.

performance/memo/README.md

@@ -4,3 +4,6 @@ Measurements of `memo`, the CLI that joins what the owner knows across the store
 
 - [lists-paging](lists-paging/README.md) — `memories`, `facts` and `actions list` past 500 rows with `--page`: a later
   page costs the same as the first, and no brief flag beyond `actions list --brief` is needed (2026-11-10).
+- [people-list](people-list/README.md) — `people list` with each person's kind kept in the store (cli-core store
+  version 62): 0.7 to 2.4 s down to 30 to 40 ms on 113,000 people, the same answers, and the kind counts in 4 ms
+  (2026-11-10).

performance/memo/people-list/README.md

@@ -0,0 +1,102 @@
+# memo people list: kinds kept in the store instead of worked out on every call
+
+## Why it was run
+
+The owner asked to make WireCat Desktop "work real fast" ("we have rather simple queries"). Desktop's page speed run
+([../../desktop/page-speed](../../desktop/page-speed/README.md)) found that `memo people list` took 3.7 to 4.2 s on a
+warm server with the owner's 113,705 people, whatever the limit, kind or order. The People page made 6 such calls:
+the list and 5 kind tabs, about 11 s for the counts alone. A CPU profile showed why: cli-core worked out every
+person's kind, message count and name order (about 1.8 s in SQLite and 1.5 s in name checks) before it applied the
+limit.
+
+The question: if each person's kind and sort keys are kept on `persons` and kept current by triggers (cli-core
+`perf/people-list`, store version 62), what does a list cost, and are the answers the same? This decides whether
+Desktop's People and Tasks pages can rely on `people list` and the new `people.counts`.
+
+## Setup
+
+- Machine: AMD Ryzen AI 9 HX 470 (24 threads), 30 GB RAM, NVMe disk, Linux 7.0, Node 24.19.0. Other agents' jobs
+  shared the machine (load average 5 to 8), so differences under about 10 ms are noise.
+- Code: before, cli-core `origin/main` at `cfdb54d07` (0.626.0); after, branch `perf/people-list` at `03f80d768` ([cli-core #2817](https://github.com/WireCatLabs/cli-core/pull/2817)). Both
+  built with `pnpm build`. The benchmark calls the store directly (`store.people.list`, `store.people.counts`), so
+  memo's process start (about 0.65 s cold, 30 to 55 ms on a warm server) is not in these numbers.
+- Data, all synthetic. [`seed.mjs`](seed.mjs) builds on [`../../people/ties/bench.mjs`](../../people/ties/bench.mjs)
+  at the owner's size: 114,700 identities and 114,721 persons (113,701 after 1,000 merges), 9,800 chats, 2,000,000
+  messages (more than half from 2,000 busy people), 64,000 emails from 6,000 mail senders (a third into a bulk mailbox, some
+  from no-reply and role addresses), meetings, an address book, CRM records, 200 tasks and 20 people added by hand.
+  Names include companies, notification senders and unreadable ones, so every kind is present.
+- Each number is the median of 7 runs after one warm-up. "first" is the warm-up. The after column ran at a load
+  average of about 7. After the upgrade the first call also
+  works out every person once.
+
+```sh
+node seed.mjs <cli-core checkout with dist> /tmp/people.db
+cp /tmp/people.db /tmp/before.db; cp /tmp/people.db /tmp/after.db
+node bench.mjs <before checkout>/dist /tmp/before.db /tmp/answers-before.json
+node bench.mjs <after checkout>/dist /tmp/after.db /tmp/answers-after.json
+cmp /tmp/answers-before.json /tmp/answers-after.json
+node after-sync.mjs <after checkout>/dist /tmp/after.db
+node write-cost.mjs <a store at version 61>; node write-cost.mjs <the same at version 62>
+cp /tmp/people.db /tmp/review.db
+node review.mjs <before checkout>/dist /tmp/review.db; node review.mjs <after checkout>/dist /tmp/review.db
+```
+
+## Results
+
+Full output: [`results.txt`](results.txt).
+
+| Call | Before | After |
+| --- | ---: | ---: |
+| `kind:person`, by messages, limit 1 | 660 ms | 40 ms |
+| `kind:person`, by messages, limit 50 | 704 ms | 42 ms |
+| `kind:person`, by messages, limit 5000 | 1,032 ms | 134 ms |
+| `kind:all`, by messages, limit 5000 | 1,433 ms | 162 ms |
+| `kind:all`, by name, limit 50 | 762 ms | 58 ms |
+| `kind:person`, by newest contact, limit 50 | 2,439 ms | 51 ms |
+| `kind:person`, by tasks, limit 50 | 785 ms | 52 ms |
+| one kind tab (organization, automated, stranger, unsorted), limit 1 | 945 to 1,192 ms | 44 to 68 ms |
+| a name search, `kind:all` | 101 ms | 64 ms |
+| `people.counts`, every kind | (5 lists, about 5 s) | 5 ms |
+| limit 50 when 500 / 2,000 / 5,000 people are left to work out | | 113 / 211 / 322 ms |
+| limit 50 after a sync of 10,000 messages from 2,000 senders (`after-sync.mjs`) | | 139 to 147 ms |
+| first list after the upgrade (works out everyone) | | 2,782 ms |
+| `people.linkProposals`, 500 of 25,000 proposals (`memo people review`, [`review.mjs`](review.mjs)) | 190 ms | 12 ms |
+
+- **Answers:** every page of 6 kinds × 4 orders × 3 offsets (0, 50, 4,950), a name search and the add-person name
+  match are byte-identical before and after (`cmp` of the two answer files).
+- **Write cost:** 50,000 message inserts in one transaction, three runs each. From 2,000 senders (a sync's usual
+  shape): 1.7 to 2.3 s before, 1.9 to 2.1 s after. From 110,000 different senders (each insert then clears a new
+  person): 1.6 to 3.3 s before, 2.3 to 3.1 s after. The machine was loaded and the ranges overlap; the triggers add
+  at most about a third to a message insert, and little when few people write.
+
+## What it showed
+
+- A list now costs what its page costs: about 35 to 70 ms for any kind and order on 113,000 people, and the limit
+  decides the rest (130 to 160 ms for 5,000 rows). The kind tabs are one `people.counts` call of 5 ms instead of five
+  lists of about a second each here, and about 3.7 s each on the owner's store.
+- Most of what is left in a small page is the page's ties (about 20 ms): their tie sets are read whole before they are
+  narrowed to the page.
+- The cost moves to the first list after a write. Each person a write touched is worked out again: about 0.15 s after
+  a sync of 10,000 messages from 2,000 senders, 0.3 s for 5,000 people. Up to 5,000 are worked out under the write
+  lock; more are worked out outside it and kept only if no other process wrote meanwhile. The first list after the
+  upgrade works out everyone once, as slow as a list was.
+- A sync that writes again what the store holds clears no kind: the triggers compare old and new values, and chat
+  member lists are now updated row by row instead of deleted and written again (a cli-core test checks this). A big
+  group's or channel's size moving by one clears no member's kind either; only a change across the small group limit
+  does.
+- `memo people review` read, parsed and resolved every link proposal before it took the page. It now pages in SQL
+  when it lists everyone's proposals; a list for one person still reads them all.
+- If the owner's store is 6 to 8 times slower than this synthetic one, as the page speed run found for the old list,
+  a page there would take about 0.2 to 0.3 s on a warm server. That is inferred, not measured.
+
+## Not measured
+
+- The owner's real store. The improvement there is inferred from the synthetic one.
+- memo's command (`memo people list`, `memo people counts`) through a process or a warm server, and Desktop's page.
+- A list while another process writes all the time and more than 5,000 people are left to work out: the list then
+  answers from memory after three tries, as slowly as before.
+- A read-only open with people left to work out, which also answers from memory.
+- The cost of the triggers on writes other than message inserts (mail, chat members, links, tasks).
+- Names whose order differs between JavaScript and SQLite: the stored order compares UTF-8 bytes, the old one UTF-16
+  code units. They disagree only when a character above U+FFFF (an emoji) meets one from U+E000 to U+FFFF at the same
+  place after folding; the synthetic names have neither.

performance/memo/people-list/after-sync.mjs

@@ -0,0 +1,30 @@
+// What the next list costs after a message sync: 10,000 new messages from 2,000 senders, committed by another
+// connection, then one `people list --limit 50`. Run on a store a list has already worked out.
+//   node after-sync.mjs <cli-core dist folder> <store file>
+import { join } from "node:path"
+import { DatabaseSync } from "node:sqlite"
+import { pathToFileURL } from "node:url"
+
+const [dist, path] = process.argv.slice(2)
+const { openStore } = await import(pathToFileURL(join(dist, "store/store.js")).href)
+const store = await openStore({ env: { ...process.env, MESSAGING_STORE: path } })
+await store.people.list({ limit: 50 })
+const writer = new DatabaseSync(path)
+writer.exec("PRAGMA foreign_keys = OFF")
+const insert = writer.prepare(
+  "INSERT INTO messages (chat_id, account_id, external_id, sender_identity_id, sent_at, text, outgoing, created_at, " +
+    "source, updated_at) VALUES (?, 1, ?, ?, ?, 'hello', 0, 0, 'sync', 0)",
+)
+for (let run = 0; run < 3; run++) {
+  writer.exec("BEGIN")
+  for (let i = 0; i < 10_000; i++) insert.run(1 + (i % 50), `sync${run}-${i}`, 3_000 + ((i * 7) % 2_000), 1.9e12 + i)
+  writer.exec("COMMIT")
+  const stale = Number(writer.prepare("SELECT count(*) AS n FROM persons WHERE kind IS NULL").get()?.n)
+  const began = performance.now()
+  await store.people.list({ limit: 50 })
+  process.stdout.write(
+    `sync ${run + 1}: ${stale} people to work out, next list ${(performance.now() - began).toFixed(0)} ms\n`,
+  )
+}
+writer.close()
+await store.close()

performance/memo/people-list/bench.mjs

@@ -0,0 +1,71 @@
+// What `memo people list` costs in cli-core on the store seed.mjs makes, and whether two builds answer the same.
+// No real data, no network, no account, no CLI run.
+//   node bench.mjs <cli-core dist folder> <store file> [answers.json]
+// With a third argument it writes every answer there, to compare two builds with `cmp`.
+import { writeFileSync } from "node:fs"
+import { join } from "node:path"
+import { performance } from "node:perf_hooks"
+import { DatabaseSync } from "node:sqlite"
+import { pathToFileURL } from "node:url"
+
+const [dist, path, answersFile] = process.argv.slice(2)
+if (!dist || !path) throw new Error("usage: node bench.mjs <cli-core dist folder> <store file> [answers.json]")
+const load = (file) => import(pathToFileURL(join(dist, file)).href)
+const { openStore } = await load("store/store.js")
+const store = await openStore({ env: { ...process.env, MESSAGING_STORE: path } })
+
+const RUNS = 7
+const say = (line) => process.stdout.write(`${line}\n`)
+const median = async (label, work) => {
+  const firstStart = performance.now()
+  await work()
+  const first = performance.now() - firstStart
+  const times = []
+  for (let run = 0; run < RUNS; run++) {
+    const began = performance.now()
+    await work()
+    times.push(performance.now() - began)
+  }
+  times.sort((a, b) => a - b)
+  say(
+    `${label.padEnd(48)} first ${first.toFixed(0).padStart(6)} ms, median ${times[RUNS >> 1].toFixed(0).padStart(5)} ms`,
+  )
+}
+
+await median("kind:person, interactions, limit 1", () => store.people.list({ limit: 1 }))
+await median("kind:person, interactions, limit 50", () => store.people.list({ limit: 50 }))
+await median("kind:person, interactions, limit 5000", () => store.people.list({ limit: 5000 }))
+await median("kind:all, interactions, limit 5000", () => store.people.list({ limit: 5000, kind: "all" }))
+await median("kind:all, name, limit 50", () => store.people.list({ limit: 50, kind: "all", sort: "name" }))
+await median("kind:person, recent, limit 50", () => store.people.list({ limit: 50, sort: "recent" }))
+await median("kind:person, tasks, limit 50", () => store.people.list({ limit: 50, sort: "tasks" }))
+for (const kind of ["organization", "automated", "stranger", "unsorted"])
+  await median(`kind:${kind}, limit 1 (a tab count)`, () => store.people.list({ limit: 1, kind }))
+await median("name search, kind:all", () => store.people.list({ limit: 50, kind: "all", query: "Garcia 12" }))
+if (store.people.counts) {
+  await median("counts, every kind", () => store.people.counts())
+  // What a sync leaves for the next list: people whose stored kind a trigger cleared.
+  const writer = new DatabaseSync(path)
+  const persons = Number(writer.prepare("SELECT count(*) AS n FROM persons").get()?.n)
+  for (const stale of [500, 2_000, 5_000])
+    await median(`limit 50, ${stale} people to work out again`, () => {
+      writer.prepare("UPDATE persons SET kind = NULL WHERE id % ? = 0").run(Math.floor(persons / stale))
+      return store.people.list({ limit: 50 })
+    })
+  writer.close()
+}
+
+if (answersFile) {
+  const answers = {}
+  for (const kind of ["person", "organization", "automated", "stranger", "unsorted", "all"])
+    for (const sort of ["interactions", "recent", "name", "tasks"])
+      for (const offset of [0, 50, 4950]) {
+        const page = await store.people.list({ limit: 50, offset, kind, sort })
+        answers[`${kind} ${sort} ${offset}`] = page
+      }
+  answers.search = await store.people.list({ limit: 50, kind: "all", query: "Garcia 12" })
+  answers.similar = await store.people.similar("ivan petrov")
+  writeFileSync(answersFile, JSON.stringify(answers, null, 1))
+}
+say(JSON.stringify((await store.people.list({ limit: 1 })).counts))
+await store.close()

performance/memo/people-list/results.txt

@@ -0,0 +1,48 @@
+# before: cli-core origin/main cfdb54d07 (0.626.0)
+kind:person, interactions, limit 1               first    686 ms, median   660 ms
+kind:person, interactions, limit 50              first    684 ms, median   704 ms
+kind:person, interactions, limit 5000            first    903 ms, median  1032 ms
+kind:all, interactions, limit 5000               first   1432 ms, median  1433 ms
+kind:all, name, limit 50                         first    748 ms, median   762 ms
+kind:person, recent, limit 50                    first   2588 ms, median  2439 ms
+kind:person, tasks, limit 50                     first    701 ms, median   785 ms
+kind:organization, limit 1 (a tab count)         first    716 ms, median   976 ms
+kind:automated, limit 1 (a tab count)            first    985 ms, median   945 ms
+kind:stranger, limit 1 (a tab count)             first    904 ms, median  1192 ms
+kind:unsorted, limit 1 (a tab count)             first    976 ms, median  1026 ms
+name search, kind:all                            first    121 ms, median   101 ms
+{"person":14902,"organization":3783,"automated":5066,"stranger":88449,"unsorted":1521}
+
+# after: cli-core perf/people-list 80a645d99 (load average about 7)
+kind:person, interactions, limit 1               first   2782 ms, median    40 ms
+kind:person, interactions, limit 50              first     42 ms, median    42 ms
+kind:person, interactions, limit 5000            first    131 ms, median   134 ms
+kind:all, interactions, limit 5000               first    174 ms, median   162 ms
+kind:all, name, limit 50                         first     50 ms, median    58 ms
+kind:person, recent, limit 50                    first     62 ms, median    51 ms
+kind:person, tasks, limit 50                     first     57 ms, median    52 ms
+kind:organization, limit 1 (a tab count)         first     44 ms, median    44 ms
+kind:automated, limit 1 (a tab count)            first     35 ms, median    46 ms
+kind:stranger, limit 1 (a tab count)             first     70 ms, median    68 ms
+kind:unsorted, limit 1 (a tab count)             first     50 ms, median    45 ms
+name search, kind:all                            first     63 ms, median    64 ms
+counts, every kind                               first      7 ms, median     5 ms
+limit 50, 500 people to work out again           first    121 ms, median   113 ms
+limit 50, 2000 people to work out again          first    229 ms, median   211 ms
+limit 50, 5000 people to work out again          first    275 ms, median   322 ms
+{"person":14902,"organization":3783,"automated":5066,"stranger":88449,"unsorted":1521}
+
+# after-sync.mjs: 10,000 messages from 2,000 senders by another connection, then a list
+sync 1: 2000 people to work out, next list 147 ms
+sync 2: 2000 people to work out, next list 146 ms
+sync 3: 2000 people to work out, next list 139 ms
+
+# answers: cmp answers-before.json answers-after.json -> identical (72 pages, a search, the name match)
+
+# write-cost.mjs, 50,000 message inserts, three runs (ms)
+SENDERS=2000   version 61: 1705 2251 1756   version 62: 1875 1861 2076
+SENDERS=110000 version 61: 1606 1664 3322, 2204 2196 2066   version 62: 2287 2285 2329, 2936 2607 3104
+
+$ node review.mjs <before>/dist review.db; node review.mjs <after>/dist review.db
+linkProposals, limit 500 of 25,000: median of 8 190 ms
+linkProposals, limit 500 of 25,000: median of 8 12 ms

performance/memo/people-list/review.mjs

@@ -0,0 +1,33 @@
+// What `memo people review --limit 500` costs in cli-core: people.linkProposals over 25,000 identity-link proposals
+// of four statuses, written straight into a store seed.mjs made (once). No real data, no account, no CLI run.
+//   node review.mjs <cli-core dist folder> <store file>
+import { join } from "node:path"
+import { DatabaseSync } from "node:sqlite"
+import { pathToFileURL } from "node:url"
+
+const [dist, path] = process.argv.slice(2)
+const writer = new DatabaseSync(path)
+if (Number(writer.prepare("SELECT count(*) AS n FROM proposed_actions WHERE kind = 'identity_link'").get()?.n) === 0)
+  writer.exec(
+    "WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 25000) " +
+      "INSERT INTO proposed_actions (kind, payload, status, proposed_by_type, proposed_by_id, created_at, updated_at) " +
+      "SELECT 'identity_link', json_object('identity', 'telegram:' || (6000 + 2 * (i % 20000)), " +
+      "'with', 'telegram:' || (6001 + 2 * (i % 20000)), " +
+      "'persons', json_array('' || (6001 + 2 * (i % 20000)), '' || (6002 + 2 * (i % 20000))), " +
+      "'sides', json_array('' || (6000 + 2 * (i % 20000)), '' || (6001 + 2 * (i % 20000))), " +
+      "'joins', 'person', 'signal', 'agent', 'score', NULL, 'evidence', json_array()), " +
+      "CASE i % 4 WHEN 0 THEN 'proposed' WHEN 1 THEN 'rejected' WHEN 2 THEN 'executed' ELSE 'approved' END, " +
+      "'bot', 3, 1794273277392 + i, 1794273277392 + i FROM n",
+  )
+writer.close()
+const { openStore } = await import(pathToFileURL(join(dist, "store/store.js")).href)
+const store = await openStore({ env: { ...process.env, MESSAGING_STORE: path } })
+const times = []
+for (let run = 0; run < 8; run++) {
+  const began = performance.now()
+  await store.people.linkProposals({ limit: 500, offset: 0 })
+  times.push(performance.now() - began)
+}
+times.sort((a, b) => a - b)
+process.stdout.write(`linkProposals, limit 500 of 25,000: median of 8 ${times[4].toFixed(0)} ms\n`)
+await store.close()