Skip to content

fix(tool): record locally executed tools in OpenCode V2 - #134

Merged
dialupdisaster merged 5 commits into
DEVtheOPS:mainfrom
pedropombeiro:fix/local-tool-telemetry
Oct 6, 2026
Merged

dialupdisaster merged 5 commits into
DEVtheOPS:mainfrom
pedropombeiro:fix/local-tool-telemetry

Conversation

@pedropombeiro

Copy link
Copy Markdown
Contributor

Description

With OpenCode V2, the plugin drops telemetry for every tool that OpenCode runs locally, such as shell, read, edit, MCP tools, Code Mode, and subagents:

  • opencode.tool.duration is never recorded.
  • No opencode.tool.* spans are created.
  • opencode.commit.count and the commit log event never fire.
  • Every tool_result log event reports duration_ms: 0.

Root cause: In V2 tool events, executed means the model provider ran the tool, not that OpenCode ran it. OpenCode publishes session.tool.called with executed: tool.providerExecuted (publish-llm-event.ts#L458-L476). It then publishes the terminal event for a locally executed tool with the same false value (#L573-L591). Only provider-hosted tools, such as OpenAI web_search or Anthropic code_execution, report executed: true (#L477-L513). handleToolCalled and finishTool gated spans, durations, and commit detection on executed, so the plugin measured only hosted tools.

Fix: A tool counts as executed once the plugin observes session.tool.called. OpenCode starts a local execution only after that event is published (step.ts#L100-L128). Calls that never ran don't publish it:

Neither case is recorded as an execution.

  • Failures after the call: A tool that fails after session.tool.called, including a permission rejection or user decline (step.ts#L199-L207), is recorded as a failed execution (success=false). Its duration includes any permission wait.
  • Provider execution: The flag is now the tool.provider_executed span attribute. It accepts both the current top-level executed and the nested provider.executed shape, which OpenCode dev uses (session-event.ts#L312-L372). That covers only the payload shape: dev also renames the events to session.next.tool.*, which this PR doesn't address.
  • Subagents: Subagent dispatch spans now start at session.tool.called instead of at the first progress event that carries a child session ID. Child-session linking is unchanged.

Tests now cover local success and failure, local git commit detection, permission rejection, malformed input without a call, provider-hosted success and failure, the nested provider.executed shape, and a subagent that fails before creating a child. Two existing tests encoded the old assumption that executed: false meant the tool didn't run, so I updated them.

I also checked it end to end with OpenCode 2.0.20 and a local OTLP/HTTP JSON sink. A read call produced a 33 ms span, histogram sample, and duration_ms. A shell call running sleep 2 recorded 2,834 ms. A git commit through shell incremented opencode.commit.count and emitted the commit log event.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Chore (dependency updates, etc.)

Checklist

  • I have read the CONTRIBUTING.md document
  • My code follows the style guidelines of this project
  • bun run lint passes with no errors
  • bun run check:jsdoc-coverage passes with no errors
  • bun run typecheck passes with no errors
  • bun test passes with no errors
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the documentation accordingly
  • My commits follow the Conventional Commits specification

Related issues

Independent of #112, which covers V1 message.part.updated timestamp restamping and orphan spans. This PR fixes V2 tool events that were never measured.

Additional context

The executed gating was introduced in #132, which treated executed: false calls as invalid. That holds for malformed calls but also matches every successful local call.

In OpenCode V2 tool events, `executed` means the model provider ran the
tool. Locally executed tools always report `executed: false`, so the
plugin skipped their spans, duration histogram, and commit detection, and
reported `duration_ms: 0` in `tool_result`.

Treat a tool as executed when the plugin observes `session.tool.called`.
Malformed-input rejections and calls cancelled before publication never
emit that event, so they are still not recorded as executions. Record
provider execution as the `tool.provider_executed` span attribute, and
accept both the top-level `executed` field and the nested
`provider.executed` shape.
@dialupdisaster

Copy link
Copy Markdown
Contributor

Non-blocking test coverage suggestion: strengthen the disabled tool-tracing test in tests/handlers/disabled.test.ts. It currently only sends session.tool.input.started, which does not create a span even when tracing is enabled.

Add a local tool sequence (input.started → called with executed: false → success) with tool tracing disabled, and assert that:

  • No tool span is created.
  • The tool duration histogram and tool_result.duration_ms still record the elapsed time.
  • For a successful shell git commit call, the commit counter and commit log event still fire.

This would protect the behavior changed here: execution metadata and command capture must happen before the tracing-disabled early return, so disabling tool spans does not also disable metrics or commit detection.

Pedro Pombeiro added 2 commits October 4, 2026 00:17
…bled

Send a complete local shell call with tool tracing disabled, and assert that the plugin creates no tool span but still records duration, the tool_result log, and the git commit counter and log.
@pedropombeiro

Copy link
Copy Markdown
Contributor Author

Thanks @dialupdisaster 🏓 Good catch: the old test only sent session.tool.input.started, so it passed whether or not tracing was enabled.

I replaced it in 23a6c22 and 7ae54b2 with a full local shell git commit call (input.started → called with executed: false → success) with tool tracing disabled. It asserts that:

  • No tool span is created after called or success.
  • The duration histogram and tool_result.duration_ms record 250 ms, measured from called.
  • The commit counter and commit log event don't fire after called, then fire exactly once after success.

I confirmed the test fails if the tracing-disabled early return moves above the metadata capture in handleToolCalled.

dialupdisaster added a commit that referenced this pull request Oct 5, 2026
## Description

Add deterministic integration tests that run the real OpenCode SDK and
telemetry plugin in isolated Bun subprocesses. A scripted
OpenAI-compatible Chat Completions endpoint drives actual tool
execution, while a temporary loopback OTLP/JSON receiver captures
exported spans, metrics, and logs. No model credentials, Docker, or
external collector are required.

- Pin the OpenCode SDK and plugin development dependencies to matching
`2.0.20` versions.
- Add seven integration cases covering session/LLM telemetry, token
attributes, trace parenting, real local `read` execution, and disabled
tool tracing.
- Include three explicit expected-failure regression cases for #134.
These assert local tool spans and duration metrics; scenario setup and
actual execution remain ordinary passing tests. Change both
`test.failing` registrations to `test` when #134 lands.
- Add `bun run test:integration` and documentation in
`tests/integration/README.md`. The existing `bun test` CI step also runs
this suite.
- Isolate home, XDG directories, project config, database, and inherited
environment; enforce a process deadline and clean up servers and
temporary files.

No production plugin code changes.

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] Documentation update
- [ ] Refactoring (no functional changes)
- [x] Chore (dependency updates, etc.)

## Checklist

- [x] I have read the [CONTRIBUTING.md](../CONTRIBUTING.md) document
- [x] My code follows the style guidelines of this project
- [x] `bun run lint` passes with no errors
- [x] `bun run check:jsdoc-coverage` passes with no errors
- [x] `bun run typecheck` passes with no errors
- [x] `bun test` passes with no errors
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the documentation accordingly
- [x] My commits follow the [Conventional
Commits](https://www.conventionalcommits.org/) specification

## Related issues

Relates to #134.

## Additional context

Validation on this branch: 198 tests pass, including the three
expected-failure cases; typecheck, lint, and JSDoc coverage pass.
Separately ran all seven integration cases against an isolated copy of
the #134 commit with expected-failure markers removed, and all seven
passed normally.

The temporary receiver covers OTLP HTTP/JSON only. gRPC/protobuf
transport coverage and additional scenarios (permissions, failures,
commits, subagents) remain follow-up work.
@dialupdisaster
dialupdisaster merged commit 129feb9 into DEVtheOPS:main Oct 6, 2026
6 of 8 checks passed
@dialupdisaster

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

dialupdisaster pushed a commit that referenced this pull request Oct 6, 2026
🤖 I have created a release *beep* *boop*
---


##
[2.0.1](v2.0.0...v2.0.1)
(2026-10-06)


### Bug Fixes

* **tool:** record locally executed tools in OpenCode V2
([#134](#134))
([129feb9](129feb9))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants