Repository navigation
feat(server): automatic memory recalls only facts a blind check calls standing - #142
Conversation
…o t3/project-memory-capture-pr-a
… t3/project-memory-capture-pr-a
Fails on the current code: candidates are saved without a check, nothing records a verdict, recall offers every saved fact, and there is no checkFactStanding operation for the eval to run. `pnpm memory-standing-check --no-check` shows the old behaviour failing the four set-aside cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… standing After extraction, one more call reads only the proposed statements, from an empty directory, and classifies each as standing, task or code. Every candidate is saved with that verdict; recall offers only standing and unchecked facts, through one SQL fragment, while search and the new Set aside filter still show the rest. A failed check saves nothing and retries the batch in 30 minutes, recording a failure after three attempts. Editing a fact's text clears a check verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estating it Recall now skips a fact the message already states (#140), so the query shares the fact's terms without repeating them in order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
…statement Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent's edit or replacement keeps a fact's verdict; a person's edit clears it. Recall's term counts and corpus size now cover only recallable facts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion A keyring login keeps its home, because Codex keys it by that path, but loses tools and developer instructions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ind evaluation Text generation offers the standing check only where a runner can run blind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion isolation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the text-generation model's provider cannot run the standing check (Grok, Muse Code, Antigravity, some OpenCode and Pi setups), every batch paid for extraction, failed the check and retried three times, saving nothing. Ask the text-generation facade whether the selected instance supports the check before extracting, and finish the batch as skipped with a reason that names the setting to change. Transient check failures keep the 30-minute retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Audit of c8d336fad5a3954162a493da387b59e4b551ab58; three reproduced defects fixed and pushed.
- Memory health now computes recall trust bands and below-threshold counts from the same recallable-fact predicate as retrieval. Saved totals still include set-aside facts.
- A provider with blind-evaluation support but no investigation support no longer incorrectly advertises review-guide generation.
- Codex keyring standing checks now fail closed when MCP discovery fails, instead of treating failed discovery as an empty server list. Ordinary text generation retains its existing fallback. The user guide no longer makes a universal repository-isolation claim.
Validation: 295 focused domain/provider tests passed. The final combined changed-file gate also passed its 110 selected tests, native build, formatting, lint, both Knip scans and GraphQL checks. The cached typecheck launcher failed before a compiler started because this environment cannot initialize its seccomp supervisor; the exact direct compiler command then passed in all 27 affected packages. All seven file hashes were unchanged throughout final validation.
I reviewed and accepted the intentional 49-byte MCP schema increase for the set-aside list filter using the repository's agent-context-increase maintainer label. No budget values or CI rules were weakened.
Remaining scope/validation limits are real: Codex keyring checks can still inherit global home instructions as documented; the current editors cannot restore a set-aside fact whose citations are unavailable; unsupported providers skip capture and Settings does not warn at selection. The private model-quality replay was not independently rerun. The test-t3-app skill requires built-in T3 Browser preview tools and says to stop UI verification when absent; those tools are absent here, so integrated UI verification could not be repeated. These limitations remain documented rather than being silently broadened into unrelated product changes.
Marked ready after publishing the fixes. Fresh CI must pass on this reviewed head before merge.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
A further integration review found an isolation edge case that must be fixed before this head is merged: if (judgeConfiguration?.source) treats an empty filtered TOML string as false. For file-backed authentication with no routing configuration, that skips creation of the isolated Codex home while the remaining path assumes isolation. null is the keyring sentinel; an empty string is valid isolated configuration. I am replacing the truthiness guard with an explicit non-null check and extending the existing standing-check regression before publishing an updated head. The previous local validation remains valid for its tested fixtures, but did not cover this empty-configuration case.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Correction to my immediately preceding comment: the suspected empty-config isolation defect is not present with the pinned TOML serializer. It serializes an empty configuration as a newline, not a byte-empty string. A new empty-routing fixture passed on the unmodified source, including the isolated-home and credential-identity checks. My earlier conclusion inferred the serializer output incorrectly. No further #142 production fix or head update is needed on that evidence; c8d336fad5a3954162a493da387b59e4b551ab58 remains the reviewed head, pending CI. The explicit non-null distinction and extra fixture coverage will be retained when integrating #142 with the summary-isolation changes in #144.
Problem
Automatic memory decides what to save while it reads the conversation, where "the PR" or "do not touch the parser" read as lasting facts. A blind audit of saved facts found about half were one task's limits or descriptions of code. Nothing ever demotes them, so they get recalled into unrelated work.
Fix
After extraction, one more call (
checkFactStanding) reads only the proposed statements, from an empty directory, and classifies each one asstanding,taskorcode. It also checks that the statement names what it refers to. The prompt is the untuned v1 from the design experiment.standingand unchecked facts. Explicit search and lists still return set-aside facts, and existing facts (no verdict) are recalled as before.resultsarray fails the call. A failed check saves nothing: the batch stays queued and retries 30 minutes later, and after 3 attempts it records "Standing check failed".t3_knowledge_mutateedit or supersede) keeps the verdict.projectKnowledgeStandingCheck. Wire changes are additive.agentContextBudget.jsonis updated to the measured value.Blindness per provider
Text generation offers the check only from a provider that can also run blind model evaluation, and the check runs with that provider's evaluation isolation:
--safe-mode, every tool disallowed, no setting sources.CODEX_HOMEholding only routing config and the login, tools and agent features off, project docs off, and any tool activity in the event stream fails the call. Exception: with a keyring,autoorephemeralcredential store, the check keeps the realCODEX_HOME, because Codex keys a keyring login by that path (the 0.169 binary uses acli|-prefixed key derived from it). That check still gets the judge flags,developer_instructions="", MCP servers off and tool rejection. What still reaches it isCODEX_HOME/AGENTS.mdand any otherconfig.tomlsetting, such as a model instructions file.codex debug prompt-inputwith canary files confirms this. Model evaluation still refuses keyring logins.SYSTEM.mdandAPPEND_SYSTEM.md. An instance whose launch args add prompts, skills or extensions does not offer the check.Proof
Failing tests first (
a9a7e167c3): a task verdict is stored set aside and is not recalled, a standing one is recalled, a failed check saves nothing and retries, explicit search still finds set-aside facts, and unchecked facts are still recalled. All fail on the previous code.Review fixes, each with a test that failed first for the stated reason:
The prompt and the JSON schema the model sees are unchanged (the schema is byte-identical).
Eval
evals/memory-standing-check(invented Acme cases), 3 samples each,--concurrency 1:--no-check(old behaviour): fails all 4 set-aside cases.Private replay of the production check on 200 freshly audited facts (not used to write any prompt), batches of 3, 3 runs per model, against the three-judge majority (129 standing, 71 not):
Without the check, saved facts in this sample are 65% standing. The admit decision flips between runs for 13–19% of facts, so a verdict is never re-run casually.
vp run check:changed origin/mainpasses.Limits
classify, and the MCP tool wording are all deferred to PR B. Until then, a set-aside fact returns to recall only when a person edits its text.Opus 5.5, Claude Code in T3 Code.
🤖 Generated with Claude Code