Skip to content

Reduce fixed injected context per session: adaptive, non-duplicative tool/prompt loading #5252

Closed

Description

@ercangorgulu-gembox

Describe the feature or problem you'd like to solve

Every interactive CLI session injects a large fixed context before the first user message (roughly 40k tokens in a comparable Copilot App session — mostly tool definitions and static system-prompt blocks, not conversation). Users have no way to see or trim what is injected, and several classes of injection are duplicated or rarely needed:

  • Tool usage prose is duplicated between the system prompt (<tools> guide) and the tool definitions themselves.
  • Long usage essays embedded in tool descriptions (e.g. a ~600-token DuckDB/SQLite guidance essay in session_store_sql) are loaded every session even when the tool is never called.
  • Skill listings inject full descriptions — dozens of skills × 50–100 tokens each — though only one or two are used per session.
  • Deferred/lazy tool reminders can list multiple overlapping toolsets from different MCP servers at once (two full browser-automation suites in one session), even when neither is used.
  • Rarely used first-party tool groups (workflow orchestration, PR review-reply, session-broadcast tools) are eager-loaded in contexts that don't need them.

Since the CLI ships as a binary, users cannot trim any of this themselves; it is a product-side lever.

Proposed solution

Adaptive, non-duplicative context injection:

  1. Single source of truth for tool guidance: keep usage prose only in the tool definitions; drop the duplicated system-prompt <tools> block (or vice versa).
  2. Load long per-tool usage guidance lazily: keep a one-line summary in the description, return the full essay in the tool result on first invocation.
  3. Cap skill-list descriptions at ~15–20 words in the injected <available_skills> block; the full text is already available when a skill is invoked.
  4. When multiple MCP servers expose overlapping capability (e.g. two browser-automation suites), collapse the deferred-tool listing to one pointer discoverable via the tool-search mechanism; ideally detect overlap automatically.
  5. Defer eager loading of rarely used tool groups (dynamic workflows, PR-lifecycle/reply tools, agent-broadcast) behind tool-search discovery, and eager-load them only in contexts that actually use them (parent agent gets task; post-review contexts get review-reply tools; etc.).
  6. Bonus: a --debug-context (or similar) breakdown so users and maintainers can see exactly what is injected and how much each block costs.

Estimated effect: ~15–20% smaller fixed overhead per session (~6–8k tokens), with no capability loss since everything stays discoverable.

Example prompts or workflows

  • Starting a quick question session (gh-style lookup): none of the workflow/MCP tools are used; the session should not pay for them.
  • First call to a lazy tool: tool-search surfaces it, and the full usage guidance arrives with the first result instead of in every session preamble.
  • Session with Playwright and Ghostlight MCP registered: only one browser toolset is advertised up front; the other is opt-in.
  • Running /security-review: its output-format contract is injected only when it runs, not in every session.
  • Long 200-message chat: freed fixed overhead extends the usable window before compaction.

Additional context

Measured from a Copilot App chat session with 48 eager tool definitions, ~19 skills, 15 configured repos, and 5 MCP servers registered; comparable structure in the CLI. The App-side equivalent request is filed in github/app (cross-linked there). Items 1–4 apply to both products.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions