Skip to content

perf: rerun the CRM relationships check, and measure tools search on task phrases - #358

Merged
leemour merged 1 commit into
mainfrom
perf/crm-rerun-tools-search
Nov 11, 2026
Merged

leemour merged 1 commit into
mainfrom
perf/crm-rerun-tools-search

Conversation

@leemour

@leemour leemour commented Nov 11, 2026

Copy link
Copy Markdown
Member

Task

Rerun the CRM relationships scenario (#315) on released b24 and memo, and measure cli-core's new tools search
(cli-core #2878) before and after.

Why

  • Check 11 failed on 2026-11-09 (a denied deal showed in an unattended search all). Its fix, cli-memo #777, shipped since.
  • Check 12 found that memo_tools_search finds only 6 of 13 commands when an agent words a task; #2878 changes the ranking.

What

  • performance/crm/relationships: rerun on b24 0.13.0 and memo 0.131.84 passes 13 of 13; check 11 holds.
    Three runner fixes, none a product bug: choose the CRM user whose deals b24 copies, the fake portal answers
    user.admin, and the owner keeps one contact that memo now hides as test data (cli-core #2862).
  • performance/agents/tools-search: 13 short and 13 task phrases over a real stdio memo mcp serve.
    Task phrases found 6 → 13 of 13, first place 6 → 9, top 3 6 → 10; short phrases unchanged (13, 9, 12).
    The after run used local builds (memo branch on cli-core main 0.643.0, unpublished); the README says to rerun it
    on the release.
  • Linked from performance/agents/README.md; CHANGELOG entries under Unreleased.

Testing

  • Both scripts ran end to end in temp folders with no keyring and no real account.
  • pnpm lint, pnpm docs:check, rumdl on the changed pages.

🤖 Generated with Claude Code

…task phrases

- performance/crm/relationships: rerun on b24 0.13.0 and memo 0.131.84 passes 13 of 13;
  check 11 holds since cli-memo #777. The runner chooses the CRM user b24 copies deals of,
  the fake portal answers user.admin, and the owner keeps one contact memo now hides as
  test data (cli-core #2862).
- performance/agents/tools-search: 13 short and 13 task phrases over a real memo MCP server.
  Ranking by matched words (cli-core #2878) finds 13 of 13 task phrases instead of 6; short
  phrases unchanged. The after run used local builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leemour
leemour merged commit f5df782 into main Nov 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant