Skip to content

agent-testing.md

Latest commit

 

History

History
174 lines (128 loc) · 7.8 KB

agent-testing.md

File metadata and controls

174 lines (128 loc) · 7.8 KB
Raw

Agent Testing Guide

This document describes how to test, evaluate, and tune .agent.md files by validating body output quality through manual testing. This is the practical equivalent of skill-testing.md applied to agents, but simpler because agents are manually invoked (no auto-triggering to test).

Why agent testing is different from skill testing

Dimension Skill Agent
Trigger mechanism description: field in SKILL.md YAML — auto-matched @agent-name mention in Copilot Chat — manual
Body loaded when? When skill description matches your prompt When user explicitly @-mentions the agent name
What to tune Description keywords for activation + body instructions Body sections (scope, handoff, modes); description for documentation only
Tool for eval loop skill-creator (optimize trigger descriptions) Manual testing (invoke with @agent-name and verify outputs)

The key insight: an agent's description: is documentation only — it does not affect invocation. Unlike skills (auto-triggered by description match), agents require explicit @agent-name mention. Use the description: field to document the agent's purpose and scope for users.

1. Recommended workflow

Since agents are invoked explicitly via @agent-name (not by description matching), skip trigger evals and focus on body quality:

  1. Create body eval cases in .github/agents/evals/<agent-name>/evals.json
  2. Start a Copilot Chat session from the repository root
  3. Invoke the agent with @agent-name and test the prompts from your evals
  4. Review output quality against expected results
  5. Edit structural sections directly in the .agent.md file (tools list, scope, handoff, modes)
  6. Re-run evals; repeat until stable

Note: The description: field in .agent.md is for documentation/help text only — it does not affect invocation. Update it to clearly state the agent's purpose and scope.

2. File layout

.github/
  agents/
    my-agent.agent.md          ← agent definition
    evals/
      my-agent/
        evals.json             ← body behavior tests
        files/                 ← fixture files referenced by evals

Store at .github/agents/evals/<agent-name>/evals.json. Same schema as skill evals:

{
  "agent_name": "my-agent",
  "evals": [
    {
      "id": "release-config-assembly",
      "prompt": "I want to set up releases for this repository. Versions are in pyproject.toml.",
      "expected_output": "Agent reads the current version from pyproject.toml, proposes the release config, lists the branch and tag naming it will use, confirms the token env var name before touching the remote.",
      "files": ["pyproject.toml"]
    },
    {
      "id": "changelog-mode-dispatch",
      "prompt": "CHANGELOG — assemble notes for 2.4.0.",
      "expected_output": "Agent loads the merged pull requests since the 2.3.0 tag, groups them by label, drafts the notes. Does NOT create the tag or push anything.",
      "files": ["CHANGELOG.md"]
    },
    {
      "id": "hotfix-scope",
      "prompt": "HOTFIX — only these 2 pull requests go into 2.3.1: #418 and #421.",
      "expected_output": "Agent scopes the release to those 2 pull requests only. Does not pick up other merged work on the branch.",
      "files": []
    }
  ]
}

4. Running body evals

When testing your agent, manually invoke it with @agent-name using the prompts from evals.json:

@my-agent I want to set up releases for this repository.
Versions are in pyproject.toml.

Verify the output against the expected_output field. Repeat for each eval case until all pass.

What to validate:

  1. All body-referenced tools are present in the frontmatter tools: list
  2. Mode dispatch (e.g., CHANGELOG, HOTFIX) activates correctly when mode keywords appear in the prompt
  3. Scope boundaries match the ## Scope and ## Does NOT sections
  4. Handoff targets are correct when the agent needs to delegate work

Tip: If an eval expects a tool that's not available, add it to the .agent.md frontmatter tools: list. If scope is wrong, edit the relevant section (## Scope, ## Does NOT, or a specific mode block).

5. Structural edits

When body evals reveal a section is wrong (wrong scope, missing tool, bad handoff), edit the .agent.md file directly:

  • Missing tool — add the tool name to the tools: list in the YAML frontmatter
  • Wrong scope boundary — update the relevant section (## Scope, ## Does NOT, or the specific mode block)
  • Broken handoff — update the ## Handoff section with the correct target agent and conditions

6. What to tune — agent-specific checklist

Check Good signal Bad signal
Scope boundaries Refuses work outside its Does-NOT list Silently attempts work outside its scope
Mode activation CHANGELOG / HOTFIX / PUBLISH activate on correct triggers Wrong mode fires, or modes don't activate
Handoff clarity Outputs correct hand-off message to the right agent Hands off to wrong agent or swallows the work
Tool completeness All tools needed by the body are in the frontmatter tools: list Body references a tool not in tools: — it will be unavailable
Description clarity Description field clearly explains agent purpose and usage Description is vague or misleading about scope

7. Description best practices

Since the description: field is documentation-only (not an activation trigger), use it to communicate the agent's purpose, domain, and key scope boundaries to users:

Good pattern:

description: >
  Release engineering for this repository: cut a release branch, assemble the changelog
  from merged pull requests, check version numbers and tags, draft the release notes.
  Use @-mention: @release-agent COMMAND.
  NOT for: writing code, reviewing diffs, CI configuration, dependency bumps.

Bad patterns:

# BAD — vague, doesn't explain domain or usage
description: >
  Helps with testing and documentation.

# BAD — talks about auto-triggering (agents use @-mention only)
description: >
  Automatically activated when you describe a release task.

  Release engineering (release branches, changelog assembly from merged pull requests,
  version numbers, tags, draft release notes, publishing checks, post-release
  verification, hotfix branches, rollback procedure). Setup: release config, credential
  checks, guided walkthrough. NOT for: writing code, reviewing diffs, CI configuration,
  dependency bumps, debugging, performance, security.

The NOT for: clause is as important as the positive terms — it prevents the agent from firing on adjacent-but-out-of-scope requests. An explicit Triggers: keyword list is not required; structured domain nouns and verbs are sufficient for the matching mechanism to work.

9. Body eval iteration loop

Repeat until all body evals pass:

  1. Invoke the agent with @agent-name using a prompt from evals.json
  2. Review output against the expected_output field
  3. If output doesn't match, identify the root cause:
    • Wrong tool? — add the missing tool to the tools: list in the frontmatter
    • Wrong scope? — update the ## Scope or ## Does NOT section
    • Mode didn't activate? — fix the mode dispatch logic (e.g., CHANGELOG, HOTFIX keywords)
    • Bad handoff? — correct the ## Handoff section
  4. Edit the .agent.md file directly with the fix
  5. Re-run that eval case
  6. Move to the next eval case; repeat until all pass

Tip: Run body evals in order (easiest first) to catch missing tools early before diving into scope or mode issues.

For the full eval methodology (subagent spawning, benchmark aggregation, the viewer), see skill-testing.md — the process is identical once the eval files are in place.