Skip to content

FEAT Preserve labeled true/false verdicts with explicit selection - #3676

Open
Aman (its-amann) wants to merge 4 commits into
microsoft:mainfrom
its-amann:feat/multi-label-true-false-2565
Open

Aman (its-amann) wants to merge 4 commits into
microsoft:mainfrom
its-amann:feat/multi-label-true-false-2565

Conversation

@its-amann

Copy link
Copy Markdown
Contributor

Description

Fixes #2565 by adding the core multi-label scoring contract.

MultiLabelTrueFalseScorer and MessageMultiLabelTrueFalseScorer preserve one true/false Score per declared category. Message pieces are aggregated within each label, and incomplete or ambiguous output is rejected before scores are saved.

TrueFalseScoreSelector projects a named verdict for attacks, boolean composition, and objective evaluation. One source call returns the selected public score and retains all labeled child scores as intermediate results. Expectations, observation links, target discovery, and blocked-judge policy are preserved.

Category queries now match JSON array members through the existing backend helpers, so the saved verdicts can be retrieved independently. Existing single-verdict scorers and the WildGuard API remain unchanged. This covers the core stage proposed in the issue; a WildGuard migration can follow separately.

This follows the design discussion in #2858 and the current contracts from #2899 and #2916. Roman Lutz (@romanlutz) Richard Lundeen (@richlundeen)

Tests and Documentation

  • 62 focused cases covering label validation, per-label aggregation, persistence and category queries, shared observations, full expectations, batching, wrappers, evaluation, and the actual attack outcome.
  • Broader scorer, memory, registry, lazy-import, and single-turn attack suite: 6,175 passed, 116 optional tests skipped.
  • New scorer modules have 99% line coverage. Ruff, targeted strict typing, applicable hooks, and lock consistency checks passed.
  • SQLite persistence and SQL Server query compilation are covered. No live model calls were needed.
  • Added and executed the documentation example: one classifier call, selected verdict False, and the other saved verdict queried without another call.
  • Wheel and source distribution builds passed, including an installed-wheel smoke test of selection and retained child score

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT Preserve multiple labeled true/false verdicts from one scorer call

1 participant