improvement(mothership): rank integration discovery by bm25 and connected services by waleedlatif1 · Pull Request #9262 · simstudioai/sim

waleedlatif1 · 2026-10-20T23:51:45Z

Summary

Integration discovery ranks by BM25 instead of substring term counts. The mothership agent searches in natural language (integrations list --query <intent>, search_integration_tools), but the old scorer had no term rarity and counted stop words and single letters. "send an email to a customer" matched 6,027 of ~6,000 operations, with gmail_send around rank 546.
Tokens come from the op name (split on _/-), service, description and parameter names, with stop words, one-character terms and plurals folded. An exact toolId still ranks first, and filters, limit, total and truncated are unchanged. A stop-word-only query matches nothing.
Services the requester has connected rank higher (×1.5), using the workspace's accessible OAuth credentials (or the org's managed ones), looked up in parallel and only for a free-text query. This only reorders: it never adds or exposes an operation.
A service filter now returns input schemas like a query does, saving the agent a round trip.

On the real registry: "send email" → gmail_send #6 (#1 with Gmail connected); "post a message to a slack channel" → slack_message #3; "list my calendar events for today" → google_calendar_list_v2 #3; "update a row in a google sheet" → google_sheets_update_v2 #3.

Type of Change

Other: discovery quality improvement

Testing

catalog.test.ts, on a 40-op fixture of real registry ops (11 email-sending services):
- ranking: connected Gmail send in the top 3; a connected service beats an equally relevant unconnected one; a stop-word-only query matches nothing; results must share a content term; an exact ID still wins;
- a workspace request looks up the requester's credentials;
- a service filter returns schemas.
Six of these fail on the old scorer. 42/42 pass in the catalog test files, and tsc --noEmit is clean. CI runs the rest.
Ranking costs about 26 ms per call over the full catalog; tokens can be cached per tool id if that matters.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-10-20T23:51:50Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 21, 2026 1:33am UTC

cubic-dev-ai

2 issues found across 2 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/mothership/integrations/application/catalog.ts">

<violation number="1" location="apps/sim/lib/mothership/integrations/application/catalog.ts:72">
P2: `singular` does not fold `-es` plurals, so `addresses`, `searches`, `branches` and `fixes` stay as `addresse`, `searche`, `branche` and `fixe` and miss their singular forms. Strip `es` after sibilant and `x`/`us` endings before the generic `s` rule.</violation>

<violation number="2" location="apps/sim/lib/mothership/integrations/application/catalog.ts:296">
P2: A failure in the credential lookup now fails the entire catalog call, even though its result only reorders results. Catch the error and fall back to no connected-service boost.</violation>
</file>

greptile-apps · 2026-10-20T23:56:09Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues remain.

Summary

Integration discovery now ranks matches with BM25 and boosts services the requester has connected.

Integration searches rank relevant tools and favor connected services.
Service-filtered integration results include each operation's input schema.

_{Reviews (7) · Last reviewed commit: "improvement(mothership): rank integratio..." · Reviewed by Greptile}

waleedlatif1 · 2026-10-21T00:03:10Z

@greptile

waleedlatif1 · 2026-10-21T00:03:11Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-21T00:03:16Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 2 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-21T00:19:57Z

@greptile

waleedlatif1 · 2026-10-21T00:19:58Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-21T00:20:04Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 2 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-21T00:37:37Z

@greptile

waleedlatif1 · 2026-10-21T01:00:47Z

@greptile

waleedlatif1 · 2026-10-21T01:00:48Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-21T01:00:54Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 2 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-21T01:17:06Z

@greptile

…cted services Free-text integration discovery now ranks with Okapi BM25 over each operation's name, service, description and parameter names, instead of substring hits. Tokens drop stop words before and after folding plurals (-ies, -es with warehouses keeping their e, -s), so natural-language intents like "send an email to a customer" match on content words. Operations of services the requester can already authenticate have their score multiplied by 1.5. Connected providers come from the authorized target workspace (accessible plus enrolled managed OAuth credentials) or, without a workspace, the requester's own org-managed credentials. The lookup only hints ranking: it never adds or reveals an operation, and a failure is logged and discovery proceeds unboosted. Sort order: exact operation ID, then boosted BM25 score, then preferred (tagged MCP) service, then name. The connected providers join the existing preferredServices tie-break in one CatalogProjectionOptions object. A service filter now counts as targeted discovery and returns input schemas.

waleedlatif1 · 2026-10-21T01:33:25Z

@greptile

waleedlatif1 · 2026-10-21T01:33:26Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-21T01:33:31Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 2 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

vercel Bot temporarily deployed to Preview October 20, 2026 23:51 Inactive

cubic-dev-ai Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts

greptile-apps Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts Outdated

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts Outdated

vercel Bot temporarily deployed to Preview October 21, 2026 00:20 Inactive

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts Outdated

vercel Bot temporarily deployed to Preview October 21, 2026 00:26 Inactive

greptile-apps Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts Outdated

vercel Bot temporarily deployed to Preview October 21, 2026 01:00 Inactive

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/integrations/application/catalog.ts

vercel Bot temporarily deployed to Preview October 21, 2026 01:14 Inactive

waleedlatif1 force-pushed the improvement/catalog-bm25-ranking branch from 2492518 to 247e3cf Compare October 21, 2026 01:33

vercel Bot temporarily deployed to Preview October 21, 2026 01:33 Inactive

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

waleedlatif1 merged commit 9c977fb into staging Oct 21, 2026
54 checks passed

waleedlatif1 deleted the improvement/catalog-bm25-ranking branch October 21, 2026 01:56

This was referenced Oct 21, 2026

fix(mothership): keep untagged mcp discovery side-effect free and the catalog resilient #9266

Merged

v0.9.30: mship improvements, mcp tool and skill discovery, docs refinements #9270

Merged

Conversation

waleedlatif1 commented Oct 20, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

waleedlatif1 commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Oct 20, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 20, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading