Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
f1b607d
FEAT: Custom ASSERT judge dimensions, presets, and repeated samples
richlundeen Nov 10, 2026
1b753ee
FIX: Keep non-default ASSERT judge settings in identity; restrict jud…
richlundeen Nov 10, 2026
54b1aba
MAINT: Simplify ASSERT judge dimensions and samples; allow technique …
richlundeen Nov 10, 2026
fe1ff0e
FEAT: Run registered attack techniques in foundry.assert_policy
richlundeen Nov 10, 2026
fb0caff
FEAT: Support ASSERT not-applicable answers and graded judge dimensions
richlundeen Nov 10, 2026
aed1b20
FEAT: assert_package technique with ASSERT's tester prompt
richlundeen Nov 10, 2026
a99189d
FIX: address #4132 review feedback
richlundeen Nov 10, 2026
a18bc4f
MAINT: typed ASSERT judgment metadata and YAML dimension framing
richlundeen Nov 11, 2026
3c26aec
FEAT: Trace-grounded ASSERT judging with OpenTelemetry execution evid…
richlundeen Nov 10, 2026
053dd5c
FIX: Order ASSERT trace steps deterministically when start times tie
richlundeen Nov 10, 2026
9780189
FIX: Address ASSERT trace evidence review feedback
richlundeen Nov 10, 2026
e2c965b
FIX: Version ASSERT trace rendering for cyclic steps
richlundeen Nov 10, 2026
4bb2163
FIX: Render ASSERT dimension trace instructions through the prompt te…
richlundeen Nov 11, 2026
a24ce22
FEAT: ASSERT headline metrics for foundry.assert_policy runs
richlundeen Nov 10, 2026
166de36
FIX: Address ASSERT metrics review feedback
richlundeen Nov 10, 2026
265a2e6
DOC: Refresh ASSERT metrics notebook output for assert_package
richlundeen Nov 10, 2026
b70cb51
FIX: Read ASSERT judgment metadata through the typed models
richlundeen Nov 11, 2026
ee00756
MAINT: merge main 1dc1d5cda into ASSERT judge dimensions (-1)
richlundeen Nov 11, 2026
9e697fd
MAINT: Merge refreshed ASSERT judge dimensions layer (main 1dc1d5cda)…
richlundeen Nov 11, 2026
78f2477
Merge commit '9e697fddfef02c1c17fb1b851cb6285c5bbc704d' into richlund…
richlundeen Nov 11, 2026
0012d16
MAINT: merge main c3cff1831 into ASSERT judge dimensions (-1)
richlundeen Nov 11, 2026
7bf65f3
MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff1831)…
richlundeen Nov 11, 2026
4848f91
Merge commit '7bf65f3969fb6cd8b2c14be28456a848baf9c370' into richlund…
richlundeen Nov 11, 2026
6932faa
FEAT: accept a supplied ASSERT objective scorer and use default scori…
richlundeen Nov 11, 2026
2804b41
DOC: note default ASSERT scoring in the launch form and the supplied …
richlundeen Nov 11, 2026
8859dea
FIX: resolve the ASSERT judge target with scoring_target and report i…
richlundeen Nov 11, 2026
f696a86
MAINT: make ASSERT judgment metadata helpers keyword-only
richlundeen Nov 11, 2026
4c8ceda
MAINT: Rebuild ASSERT trace evidence layer on the supplied-scorer layer
richlundeen Nov 11, 2026
06ccdf2
MAINT: Exempt the OpenTelemetry span processor callback from keyword-…
richlundeen Nov 11, 2026
b5e5931
Merge published ASSERT metrics layer onto the rebuilt trace evidence …
richlundeen Nov 11, 2026
cee992a
FIX: Identify ASSERT runs by registry name and keep supplied scorers …
richlundeen Nov 11, 2026
880a61f
FIX: Use only proven ASSERT provenance for metrics policies and dimen…
richlundeen Nov 11, 2026
145b788
FIX: Require built-in ASSERT judge evidence for metrics policies
richlundeen Nov 11, 2026
d0b7c2a
FIX: Stop attributing ASSERT violations to the mutable assigned category
richlundeen Nov 11, 2026
b2a71e2
Merge main into ASSERT headline metrics
richlundeen Nov 17, 2026
82bddb0
FIX: Simplify ASSERT analytics around saved policy evidence
richlundeen Nov 17, 2026
9a9ddcd
FIX: Keep composed objectives separate from ASSERT judgments
richlundeen Nov 17, 2026
b05a16c
FIX: Preserve historical native ASSERT dimension metadata
richlundeen Nov 17, 2026
39bdc50
FIX: Keep ASSERT metrics columns readable on narrow screens
richlundeen Nov 17, 2026
6a3988c
TEST: Cover persisted ASSERT results in headline analytics
richlundeen Nov 17, 2026
165a2d4
Merge selected main into ASSERT headline metrics
richlundeen Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions doc/code/analytics/0_attack_results.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,34 @@ supported. The reported difference is descriptive and does not establish that
the attack caused an outcome to change. No confidence interval is provided,
because small and constant paired samples need careful uncertainty treatment.

### ASSERT policy metrics

`compute_assert_policy_metrics_async(result)` reports **impermissible behavior
violated** and **permissible behavior violated** for a `foundry.assert_policy` run.
It counts the same latest execution units as `compute_scenario_statistics`,
separately for each technique and the baseline. Each conversation counts once
per stance the judge found relevant, as violated if any relevant category of that
stance was violated, and as not applicable if none was relevant.

Category relevance comes from `AssertCategoryJudgment`; permissibility comes
from the `ViolatesAssertPolicy` in the policy score's saved `scored_expectation`.
Analytics never rereads today's seed datasets. Scores without that recorded
policy or per-category judgments are counted as exclusions, not guessed from
their generated category. A verdict that contradicts its category judgments
is rejected.

The result includes independent dimension rates, per-policy category rates,
and exclusion counts. N/A and missing verdicts stay outside rate denominators.
Grade counts and medians remain separate for each recorded scale. Rates use
ASSERT's 90% Wilson interval and flag fewer than five applicable conversations.
A supplied unmarked or composing objective scorer is reported as `objective`;
its underlying ASSERT judgments remain independent dimension results.

The run page exposes the same result through
`GET /api/analytics/assert?scenario_result_id=<run-id>`. The filter selects a
saved run, not a scenario catalog entry. See the
[Foundry notebook](../../scanner/foundry.ipynb) for an executed example.

### Scenario producer accounting

Scenario counts also expose `producer_counts`, separating target-facing,
Expand Down
Loading
Loading