Repository navigation
feat(evaluations)!: promote Evaluations from Code to core - #235
Merged
Merged
Conversation
Export the Evaluations from Code names from launchdarkly_ai_server. Remove launchdarkly_ai_server.experimental.evaluations. Report the usage helper as client.initEvaluations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
donei003
force-pushed
the
feat/promote-evaluations-to-core
branch
from
October 28, 2026 17:02
f3586b7 to
43ecea8
Compare
5 of 6 tasks
…ons-to-core # Conflicts: # packages/client/tests/test_evaluations_run.py
…ons-to-core # Conflicts: # packages/client/api-report/launchdarkly_ai_server.experimental.evaluations.txt
…ons-to-core # Conflicts: # packages/client/README.md
aknight-ld
approved these changes
Oct 28, 2026
…ons-to-core # Conflicts: # packages/client/api-report/launchdarkly_ai_server.experimental.evaluations.txt
4 of 5 tasks
donei003
added a commit
to launchdarkly/js-ai-sdk
that referenced
this pull request
Oct 29, 2026
Promotes Evaluations from Code from `@launchdarkly/ai-server/experimental` to the package root. This follows the Agent Skills promotion in #159. > **Draft.** This PR releases together with launchdarkly/python-ai-sdk#235. A Jira ticket is not linked yet. ## Migration Change the import path. The names, signatures and behavior do not change. ```ts // before import { EvalTool, initEvaluations } from '@launchdarkly/ai-server/experimental'; // after import { EvalTool, initEvaluations } from '@launchdarkly/ai-server'; ``` Use `@launchdarkly/ai-node` in place of `@launchdarkly/ai-node/experimental` in the same way. This is a breaking change (`!`). It bumps the minor version while the package is 0.x. ## What changed - **Root exports.** The 17 Evaluations from Code names (values and types) are exported from the `@launchdarkly/ai-server` root with the same spelling. No name collides with a current root export. `@launchdarkly/ai-node` gets them through its existing `export *`. - **Experimental entry point.** `src/experimental.ts` keeps its header and exports nothing. The `./experimental` subpath stays in both packages, so a future experimental feature has a place to go. An old import now gives a missing name, not a missing module. - **Usage ID.** `initEvaluations` now reports `client.initEvaluations`, not `experimental.initEvaluations`. Queries that count the old ID must also count the new one. - **API surface.** The names are added to `api-surface/ai-server.txt` as `+` and marked `-` in `api-surface/ai-server.experimental.txt`. Both API Extractor reports are regenerated. The experimental report is now empty. - **API report warnings.** The root report now has 12 `ae-forgotten-export` warnings, up from 7. The new ones are evaluations types that are not exported themselves, for example `EvalToolImplementation`, `ScorerFn` and `Transport`. The experimental report had the same warnings before, so nothing new is exposed. - **Changelog.** The three `evaluations` rows are removed from `changelog-sections`, so `feat(evaluations):` lands under Features. The `AGENTS.md` scope examples now use a placeholder scope. - **Docs.** Both READMEs and both `agents.md` files no longer call Evaluations from Code experimental. The generated export blocks in `packages/client/agents.md` are regenerated from `index.ts` and `experimental.ts`. The README anchor changed from `#evaluations-from-code-experimental` to `#evaluations-from-code`. ## Tests - The package export tests check that the experimental entry point exports no value, that the root snapshot includes the evaluations values, and that every evaluations name is in the root API report and none is in the experimental one. - The ai-node re-export test checks that the evaluations values are on the root as the same references and not on `/experimental`. - The evaluations tests import from the root. `sdk-usage.test.ts` expects `client.initEvaluations`. - The integration fixtures check that `initEvaluations` is on both roots and absent from both `/experimental` subpaths, at run time (CJS and ESM) and at type-check time. ## Checks - [x] `yarn code:check`, `yarn build`, `yarn typecheck`: pass - [x] `yarn test`: pass (client: 1929 passed, 11 skipped) - [x] `yarn test:integration`: 36 passed - [x] `yarn api:surface`, `yarn test:api-surface`, `yarn test:api-breaking`, `yarn lint:pkg`, `yarn api:check`: pass - [x] `node scripts/api-breaking.mjs pr --base origin/main`: no breaking changes to the public API - [ ] Run one evaluation end to end against staging with the root import ## Conflicts The open PR #175 edits evaluations code. Merge it first, or rebase it after this PR. ## Related PRs Releases together with: - launchdarkly/python-ai-sdk#235 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > **Evaluations from Code** is promoted from `@launchdarkly/ai-server/experimental` (and `@launchdarkly/ai-node/experimental`) to the **package root**, mirroring the earlier Agent Skills promotion. Consumers should import names like `initEvaluations`, `EvalTool`, `Judge`, and `Scorer` from `@launchdarkly/ai-server` or `@launchdarkly/ai-node` instead of the `/experimental` subpath; behavior and signatures are unchanged. > > The experimental barrel now exports nothing (`export {}`), while the `./experimental` subpath remains for future features. Public API surface lists, API Extractor reports, and `release-please-config.json` are updated accordingly, and `initEvaluations` reports usage as **`client.initEvaluations`** rather than `experimental.initEvaluations`. Docs, `AGENTS.md`, and tests (including CommonJS/webpack integration fixtures) assert evaluations live on the root only. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 591d138. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/launchdarkly/js-ai-sdk/pull/181" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes Evaluations from Code from
launchdarkly_ai_server.experimental.evaluationsto the package root. This follows the Agent Skills promotion in #210.Migration
Import Evaluations from Code names from
launchdarkly_ai_serverinstead oflaunchdarkly_ai_server.experimental.evaluations. The names, signatures and behavior do not change.This is a breaking change (
!). It bumps the minor version while the package is 0.x.What changed
experimental/evaluations.pyexported are now inlaunchdarkly_ai_server.__all__, with the same spelling. No name collides with a current root export.launchdarkly-ai-pythongets the names through its existingimport *.experimental/evaluations.pyis deleted.experimental/__init__.pystays as a docstring only, so a future experimental feature has a place to go.init_evaluationsnow reportsclient.initEvaluations, notexperimental.initEvaluations. Queries that count the old ID must also count the new one.make api-report. The names are inlaunchdarkly_ai_server.txt, andlaunchdarkly_ai_server.experimental.evaluations.txtis deleted.evaluationsrows are removed fromchangelog-sections, sofeat(evaluations):lands under Features. TheAGENTS.mdscope examples now use a placeholder scope.launchdarkly-ai-pythonREADME,packages/client/agents.mdandAGENTS.mdno longer call Evaluations from Code experimental. The README heading anchor changed from#run-an-evaluation-from-code-experimentalto#run-an-evaluation-from-code.Tests
TestPackageExportschecks that every evaluations name is on the root and thatexperimental.evaluationsno longer imports.test_public_surface.pyadds the 14 names to the root snapshot.test_reexports.pychecks that the barrel re-exports the names.test_core_does_not_import_experimental_codenow checks onlylaunchdarkly_ai_server.experimental, because the root loadsevaluations. A new test checks thatlifecycleimports no evaluations module.test_sdk_usage.pyexpectsclient.initEvaluations.Import time
import launchdarkly_ai_servernow loads theevaluationspackage. Measured withpython -X importtime, it adds about 33 ms to a root import of about 130 ms. The largest part isevaluations.api.Checks
make lint,make format-check,make typecheck,make api-report-check: passpython -m scripts.api_breaking pr --base origin/main: no breaking changes to the public APImake test: 3640 passed, 18 skippedConflicts
The open evaluations PRs #228, #227, #143, #135 and #92 edit the same test files. Merge them first, or rebase them after this PR.
Related PRs
Releases together with:
🤖 Generated with Claude Code
Note
Overview
Evaluations from Code moves from
launchdarkly_ai_server.experimental.evaluationsto thelaunchdarkly_ai_serverpackage root (and throughlaunchdarkly-ai-pythonre-exports). The experimental shim is removed; callers should importinit_evaluations,EvalTool,Judge,Scorer, and the rest from the root instead.The public API contract shifts accordingly: generated
launchdarkly_ai_server.txtnow lists the evaluations surface, andlaunchdarkly_ai_server.experimental.evaluations.txtis deleted.init_evaluationsusage telemetry changes fromexperimental.initEvaluationstoclient.initEvaluations.Release/changelog policy is updated:
evaluationsis no longer an Experimental scope inrelease-please-config.json, andAGENTS.md/ READMEs describe Evaluations (and Skills) as core, with docs and examples using root imports and a non-experimental README anchor.Tests and snapshots (
test_public_surface,TestPackageExports,test_reexports, lifecycle import guards) enforce the new export layout and thatexperimental.evaluationscannot be imported.Reviewed by Cursor Bugbot for commit 3ee3612. Bugbot is set up for automated code reviews on this repo. Configure here.