Skip to content

feat(server): automatic memory recalls only facts a blind check calls standing - #142

Merged
ScriptedAlchemy merged 17 commits into
mainfrom
t3/project-memory-capture-pr-a
Nov 17, 2026
Merged

ScriptedAlchemy merged 17 commits into
mainfrom
t3/project-memory-capture-pr-a

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Problem

Automatic memory decides what to save while it reads the conversation, where "the PR" or "do not touch the parser" read as lasting facts. A blind audit of saved facts found about half were one task's limits or descriptions of code. Nothing ever demotes them, so they get recalled into unrelated work.

Fix

After extraction, one more call (checkFactStanding) reads only the proposed statements, from an empty directory, and classifies each one as standing, task or code. It also checks that the statement names what it refers to. The prompt is the untuned v1 from the design experiment.

  • Every candidate is saved, with its verdict (migration 115). Automatic recall goes through one SQL fragment and offers only standing and unchecked facts. Explicit search and lists still return set-aside facts, and existing facts (no verdict) are recalled as before.
  • Recall's term statistics (the per-project term counts from migration 114 and the corpus size) count exactly the recallable facts. Migration 115 redefines those triggers and the partial index with the same condition, so set-aside facts never change term weights. No existing row has a verdict, so no rebuild is needed.
  • Each answer entry decodes on its own. A missing, repeated or malformed entry, or a standing statement that is not self-contained, sets aside only that statement. A reply with no usable results array fails the call. A failed check saves nothing: the batch stays queued and retries 30 minutes later, and after 3 attempts it records "Standing check failed".
  • Verdicts are written once at capture, and a duplicate keeps the existing fact's verdict. When a person edits a fact's text, its check verdict is cleared and the fact can be recalled again. An agent's edit or replacement (t3_knowledge_mutate edit or supersede) keeps the verdict.
  • Clients: the fact detail shows the verdict line, and a Set aside filter appears on web and mobile only when the server advertises projectKnowledgeStandingCheck. Wire changes are additive.
  • The MCP tool list grows 49 bytes (the new list filter); agentContextBudget.json is updated to the measured value.

Blindness per provider

Text generation offers the check only from a provider that can also run blind model evaluation, and the check runs with that provider's evaluation isolation:

  • Claude: --safe-mode, every tool disallowed, no setting sources.
  • Codex: the evaluator's isolation: a separate CODEX_HOME holding only routing config and the login, tools and agent features off, project docs off, and any tool activity in the event stream fails the call. Exception: with a keyring, auto or ephemeral credential store, the check keeps the real CODEX_HOME, because Codex keys a keyring login by that path (the 0.169 binary uses a cli|-prefixed key derived from it). That check still gets the judge flags, developer_instructions="", MCP servers off and tool rejection. What still reaches it is CODEX_HOME/AGENTS.md and any other config.toml setting, such as a model instructions file. codex debug prompt-input with canary files confirms this. Model evaluation still refuses keyring logins.
  • OpenCode (1.x and 2.x): its own server process with the blind-evaluation environment (routing config only, project and Claude Code config off), all tools denied, and tool activity fails the call. 1.x ignores a selected custom agent. With a configured server URL the check is not offered.
  • Pi: no tools, extensions, context files, skills or prompt templates. Explicit system prompts suppress the global SYSTEM.md and APPEND_SYSTEM.md. An instance whose launch args add prompts, skills or extensions does not offer the check.
  • Cursor: no tools, no setting sources, an empty workspace, and tool activity fails the call.
  • Grok, Muse Code, Antigravity: no blind mode, so they do not offer the check (see Limits for what automatic memory does then).
  • Kiro, fx and ACP Registry have no text generation. Model APIs are not used by automatic memory (extraction and the check resolve only provider instances).

Proof

  • Failing tests first (a9a7e167c3): a task verdict is stored set aside and is not recalled, a standing one is recalled, a failed check saves nothing and retries, explicit search still finds set-aside facts, and unchecked facts are still recalled. All fail on the previous code.

  • Review fixes, each with a test that failed first for the stated reason:

    • A reply with one malformed entry sets aside only that statement instead of failing the batch.
    • An agent's edit and an agent's supersede keep a set-aside fact out of recall, while a person's edit makes it recallable.
    • Recall's term counts and corpus size stay right through insert, verdict change, edit, supersede, forget and delete (migration test).
    • Codex, OpenCode 1.x and 2.x, Pi and Cursor launch the check isolated and reject tool activity. A Codex keyring login keeps its home without developer instructions or MCP servers.
    • With a provider that cannot run the check, a settled run with learnable content makes no model call, finishes as skipped with the reason above, and leaves nothing queued for retry. The transient-failure test still shows 3 attempts.
  • The prompt and the JSON schema the model sees are unchanged (the schema is byte-identical).

  • Eval evals/memory-standing-check (invented Acme cases), 3 samples each, --concurrency 1:

    • --no-check (old behaviour): fails all 4 set-aside cases.
    • morph-kimik31, morph-dsv41flash and morph-dsv41flash-1017: 6/6 cases, 18/18 samples each. Rerun on morph-kimik31 after the review fixes: 6/6, 18/18.
  • Private replay of the production check on 200 freshly audited facts (not used to write any prompt), batches of 3, 3 runs per model, against the three-judge majority (129 standing, 71 not):

    model standing kept non-standing admitted precision
    morph-kimik31 76–78% (mean 76%) 6–15% (mean 9%) 90–96% (mean 94%)
    morph-dsv41flash 74–80% (mean 78%) 11–13% (mean 12%) 91–93% (mean 92%)
    morph-dsv41flash-1017 74–80% (mean 77%) 10–13% (mean 11%) 91–94% (mean 93%)

    Without the check, saved facts in this sample are 65% standing. The admit decision flips between runs for 13–19% of facts, so a verdict is never re-run casually.

  • vp run check:changed origin/main passes.

Limits

  • The sweep of existing facts, Keep and Set aside actions, classify, and the MCP tool wording are all deferred to PR B. Until then, a set-aside fact returns to recall only when a person edits its text.
  • A set-aside fact whose cited thread has been deleted cannot be edited or superseded ("The selected source is unavailable or has changed"), because web and mobile resend the fact's citations and the server checks them. For those facts there is no way back in this PR, only Forget.
  • On Codex with a keyring login, the check is not fully blind (see above).
  • When the text-generation model's provider cannot run the check (Grok, Muse Code, Antigravity, OpenCode with a configured server URL, Pi with prompt, skill or extension launch args), automatic memory saves nothing. Each batch finishes as skipped before extraction, with no model call and no retry, and Memory health reads "Last run skipped. The text generation model's provider can't check memories. Choose another model in Settings → Threads → Text generation." A check that fails for another reason (timeout, rate limit, exhausted allowance) still retries after 30 minutes, up to 3 attempts. Settings gives no hint when you pick such a model: the text-generation model row validates extraction but not the check.
  • The check is a noise filter, not a guarantee. Morph-dsv41flash in particular admits slightly more than the design's 10% target.

Opus 5.5, Claude Code in T3 Code.

🤖 Generated with Claude Code

ScriptedAlchemy and others added 8 commits November 16, 2026 19:06
Fails on the current code: candidates are saved without a check, nothing
records a verdict, recall offers every saved fact, and there is no
checkFactStanding operation for the eval to run. `pnpm memory-standing-check
--no-check` shows the old behaviour failing the four set-aside cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… standing

After extraction, one more call reads only the proposed statements, from an
empty directory, and classifies each as standing, task or code. Every
candidate is saved with that verdict; recall offers only standing and
unchecked facts, through one SQL fragment, while search and the new Set aside
filter still show the rest. A failed check saves nothing and retries the batch
in 30 minutes, recording a failure after three attempts. Editing a fact's text
clears a check verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estating it

Recall now skips a fact the message already states (#140), so the query shares
the fact's terms without repeating them in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: c8d336f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

ScriptedAlchemy and others added 8 commits November 16, 2026 21:32
…statement

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent's edit or replacement keeps a fact's verdict; a person's edit clears it.
Recall's term counts and corpus size now cover only recallable facts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion

A keyring login keeps its home, because Codex keys it by that path, but loses
tools and developer instructions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ind evaluation

Text generation offers the standing check only where a runner can run blind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion isolation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the text-generation model's provider cannot run the standing check
(Grok, Muse Code, Antigravity, some OpenCode and Pi setups), every batch
paid for extraction, failed the check and retried three times, saving
nothing. Ask the text-generation facade whether the selected instance
supports the check before extracting, and finish the batch as skipped
with a reason that names the setting to change. Transient check failures
keep the 30-minute retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review November 17, 2026 16:28

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Audit of c8d336fad5a3954162a493da387b59e4b551ab58; three reproduced defects fixed and pushed.

  • Memory health now computes recall trust bands and below-threshold counts from the same recallable-fact predicate as retrieval. Saved totals still include set-aside facts.
  • A provider with blind-evaluation support but no investigation support no longer incorrectly advertises review-guide generation.
  • Codex keyring standing checks now fail closed when MCP discovery fails, instead of treating failed discovery as an empty server list. Ordinary text generation retains its existing fallback. The user guide no longer makes a universal repository-isolation claim.

Validation: 295 focused domain/provider tests passed. The final combined changed-file gate also passed its 110 selected tests, native build, formatting, lint, both Knip scans and GraphQL checks. The cached typecheck launcher failed before a compiler started because this environment cannot initialize its seccomp supervisor; the exact direct compiler command then passed in all 27 affected packages. All seven file hashes were unchanged throughout final validation.

I reviewed and accepted the intentional 49-byte MCP schema increase for the set-aside list filter using the repository's agent-context-increase maintainer label. No budget values or CI rules were weakened.

Remaining scope/validation limits are real: Codex keyring checks can still inherit global home instructions as documented; the current editors cannot restore a set-aside fact whose citations are unavailable; unsupported providers skip capture and Settings does not warn at selection. The private model-quality replay was not independently rerun. The test-t3-app skill requires built-in T3 Browser preview tools and says to stop UI verification when absent; those tools are absent here, so integrated UI verification could not be repeated. These limitations remain documented rather than being silently broadened into unrelated product changes.

Marked ready after publishing the fixes. Fresh CI must pass on this reviewed head before merge.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A further integration review found an isolation edge case that must be fixed before this head is merged: if (judgeConfiguration?.source) treats an empty filtered TOML string as false. For file-backed authentication with no routing configuration, that skips creation of the isolated Codex home while the remaining path assumes isolation. null is the keyring sentinel; an empty string is valid isolated configuration. I am replacing the truthiness guard with an explicit non-null check and extending the existing standing-check regression before publishing an updated head. The previous local validation remains valid for its tested fixtures, but did not cover this empty-configuration case.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my immediately preceding comment: the suspected empty-config isolation defect is not present with the pinned TOML serializer. It serializes an empty configuration as a newline, not a byte-empty string. A new empty-routing fixture passed on the unmodified source, including the isolated-home and credential-identity checks. My earlier conclusion inferred the serializer output incorrectly. No further #142 production fix or head update is needed on that evidence; c8d336fad5a3954162a493da387b59e4b551ab58 remains the reviewed head, pending CI. The explicit non-null distinction and extra fixture coverage will be retained when integrating #142 with the summary-isolation changes in #144.

@ScriptedAlchemy
ScriptedAlchemy merged commit 283b5f4 into main Nov 17, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant