Skip to content

feat(evaluations)!: promote Evaluations from Code to core - #235

Merged
donei003 merged 5 commits into
mainfrom
feat/promote-evaluations-to-core
Oct 29, 2026
Merged

donei003 merged 5 commits into
mainfrom
feat/promote-evaluations-to-core

Conversation

@donei003

@donei003 donei003 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Promotes Evaluations from Code from launchdarkly_ai_server.experimental.evaluations to the package root. This follows the Agent Skills promotion in #210.

Draft. This PR releases together with launchdarkly/js-ai-sdk#181. A Jira ticket is not linked yet.

Migration

Import Evaluations from Code names from launchdarkly_ai_server instead of launchdarkly_ai_server.experimental.evaluations. The names, signatures and behavior do not change.

# before
from launchdarkly_ai_server.experimental.evaluations import EvalTool, init_evaluations
# after
from launchdarkly_ai_server import EvalTool, init_evaluations

This is a breaking change (!). It bumps the minor version while the package is 0.x.

What changed

  • Root exports. The 14 names that experimental/evaluations.py exported are now in launchdarkly_ai_server.__all__, with the same spelling. No name collides with a current root export. launchdarkly-ai-python gets the names through its existing import *.
  • Experimental module removed. experimental/evaluations.py is deleted. experimental/__init__.py stays as a docstring only, so a future experimental feature has a place to go.
  • Usage ID. init_evaluations now reports client.initEvaluations, not experimental.initEvaluations. Queries that count the old ID must also count the new one.
  • API reports. Regenerated with make api-report. The names are in launchdarkly_ai_server.txt, and launchdarkly_ai_server.experimental.evaluations.txt is deleted.
  • Changelog. The three evaluations rows are removed from changelog-sections, so feat(evaluations): lands under Features. The AGENTS.md scope examples now use a placeholder scope.
  • Docs. The client README, the launchdarkly-ai-python README, packages/client/agents.md and AGENTS.md no longer call Evaluations from Code experimental. The README heading anchor changed from #run-an-evaluation-from-code-experimental to #run-an-evaluation-from-code.

Tests

  • TestPackageExports checks that every evaluations name is on the root and that experimental.evaluations no longer imports.
  • test_public_surface.py adds the 14 names to the root snapshot.
  • test_reexports.py checks that the barrel re-exports the names.
  • test_core_does_not_import_experimental_code now checks only launchdarkly_ai_server.experimental, because the root loads evaluations. A new test checks that lifecycle imports no evaluations module.
  • test_sdk_usage.py expects client.initEvaluations.

Import time

import launchdarkly_ai_server now loads the evaluations package. Measured with python -X importtime, it adds about 33 ms to a root import of about 130 ms. The largest part is evaluations.api.

Checks

  • make lint, make format-check, make typecheck, make api-report-check: pass
  • python -m scripts.api_breaking pr --base origin/main: no breaking changes to the public API
  • make test: 3640 passed, 18 skipped
  • Run one evaluation end to end against staging with the root import

Conflicts

The open evaluations PRs #228, #227, #143, #135 and #92 edit the same test files. Merge them first, or rebase them after this PR.

Related PRs

Releases together with:

🤖 Generated with Claude Code


Note

Overview
Evaluations from Code moves from launchdarkly_ai_server.experimental.evaluations to the launchdarkly_ai_server package root (and through launchdarkly-ai-python re-exports). The experimental shim is removed; callers should import init_evaluations, EvalTool, Judge, Scorer, and the rest from the root instead.

The public API contract shifts accordingly: generated launchdarkly_ai_server.txt now lists the evaluations surface, and launchdarkly_ai_server.experimental.evaluations.txt is deleted. init_evaluations usage telemetry changes from experimental.initEvaluations to client.initEvaluations.

Release/changelog policy is updated: evaluations is no longer an Experimental scope in release-please-config.json, and AGENTS.md / READMEs describe Evaluations (and Skills) as core, with docs and examples using root imports and a non-experimental README anchor.

Tests and snapshots (test_public_surface, TestPackageExports, test_reexports, lifecycle import guards) enforce the new export layout and that experimental.evaluations cannot be imported.

Reviewed by Cursor Bugbot for commit 3ee3612. Bugbot is set up for automated code reviews on this repo. Configure here.

Export the Evaluations from Code names from launchdarkly_ai_server.
Remove launchdarkly_ai_server.experimental.evaluations.
Report the usage helper as client.initEvaluations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ons-to-core

# Conflicts:
#	packages/client/tests/test_evaluations_run.py
…ons-to-core

# Conflicts:
#	packages/client/api-report/launchdarkly_ai_server.experimental.evaluations.txt
…ons-to-core

# Conflicts:
#	packages/client/README.md
@donei003
donei003 marked this pull request as ready for review October 28, 2026 20:01
@donei003
donei003 requested a review from a team as a code owner October 28, 2026 20:01

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

…ons-to-core

# Conflicts:
#	packages/client/api-report/launchdarkly_ai_server.experimental.evaluations.txt
donei003 added a commit to launchdarkly/js-ai-sdk that referenced this pull request Oct 29, 2026
Promotes Evaluations from Code from
`@launchdarkly/ai-server/experimental` to the package root. This follows
the Agent Skills promotion in #159.

> **Draft.** This PR releases together with
launchdarkly/python-ai-sdk#235. A Jira ticket is not linked yet.

## Migration

Change the import path. The names, signatures and behavior do not
change.

```ts
// before
import { EvalTool, initEvaluations } from '@launchdarkly/ai-server/experimental';
// after
import { EvalTool, initEvaluations } from '@launchdarkly/ai-server';
```

Use `@launchdarkly/ai-node` in place of
`@launchdarkly/ai-node/experimental` in the same way.

This is a breaking change (`!`). It bumps the minor version while the
package is 0.x.

## What changed

- **Root exports.** The 17 Evaluations from Code names (values and
types) are exported from the `@launchdarkly/ai-server` root with the
same spelling. No name collides with a current root export.
`@launchdarkly/ai-node` gets them through its existing `export *`.
- **Experimental entry point.** `src/experimental.ts` keeps its header
and exports nothing. The `./experimental` subpath stays in both
packages, so a future experimental feature has a place to go. An old
import now gives a missing name, not a missing module.
- **Usage ID.** `initEvaluations` now reports `client.initEvaluations`,
not `experimental.initEvaluations`. Queries that count the old ID must
also count the new one.
- **API surface.** The names are added to `api-surface/ai-server.txt` as
`+` and marked `-` in `api-surface/ai-server.experimental.txt`. Both API
Extractor reports are regenerated. The experimental report is now empty.
- **API report warnings.** The root report now has 12
`ae-forgotten-export` warnings, up from 7. The new ones are evaluations
types that are not exported themselves, for example
`EvalToolImplementation`, `ScorerFn` and `Transport`. The experimental
report had the same warnings before, so nothing new is exposed.
- **Changelog.** The three `evaluations` rows are removed from
`changelog-sections`, so `feat(evaluations):` lands under Features. The
`AGENTS.md` scope examples now use a placeholder scope.
- **Docs.** Both READMEs and both `agents.md` files no longer call
Evaluations from Code experimental. The generated export blocks in
`packages/client/agents.md` are regenerated from `index.ts` and
`experimental.ts`. The README anchor changed from
`#evaluations-from-code-experimental` to `#evaluations-from-code`.

## Tests

- The package export tests check that the experimental entry point
exports no value, that the root snapshot includes the evaluations
values, and that every evaluations name is in the root API report and
none is in the experimental one.
- The ai-node re-export test checks that the evaluations values are on
the root as the same references and not on `/experimental`.
- The evaluations tests import from the root. `sdk-usage.test.ts`
expects `client.initEvaluations`.
- The integration fixtures check that `initEvaluations` is on both roots
and absent from both `/experimental` subpaths, at run time (CJS and ESM)
and at type-check time.

## Checks

- [x] `yarn code:check`, `yarn build`, `yarn typecheck`: pass
- [x] `yarn test`: pass (client: 1929 passed, 11 skipped)
- [x] `yarn test:integration`: 36 passed
- [x] `yarn api:surface`, `yarn test:api-surface`, `yarn
test:api-breaking`, `yarn lint:pkg`, `yarn api:check`: pass
- [x] `node scripts/api-breaking.mjs pr --base origin/main`: no breaking
changes to the public API
- [ ] Run one evaluation end to end against staging with the root import

## Conflicts

The open PR #175 edits evaluations code. Merge it first, or rebase it
after this PR.

## Related PRs

Releases together with:
- launchdarkly/python-ai-sdk#235

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Evaluations from Code** is promoted from
`@launchdarkly/ai-server/experimental` (and
`@launchdarkly/ai-node/experimental`) to the **package root**, mirroring
the earlier Agent Skills promotion. Consumers should import names like
`initEvaluations`, `EvalTool`, `Judge`, and `Scorer` from
`@launchdarkly/ai-server` or `@launchdarkly/ai-node` instead of the
`/experimental` subpath; behavior and signatures are unchanged.
> 
> The experimental barrel now exports nothing (`export {}`), while the
`./experimental` subpath remains for future features. Public API surface
lists, API Extractor reports, and `release-please-config.json` are
updated accordingly, and `initEvaluations` reports usage as
**`client.initEvaluations`** rather than `experimental.initEvaluations`.
Docs, `AGENTS.md`, and tests (including CommonJS/webpack integration
fixtures) assert evaluations live on the root only.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
591d138. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/launchdarkly/js-ai-sdk/pull/181"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@donei003
donei003 merged commit 0bc43a5 into main Oct 29, 2026
8 checks passed
@donei003
donei003 deleted the feat/promote-evaluations-to-core branch October 29, 2026 06:41
@github-actions github-actions Bot mentioned this pull request Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants