FEAT Preserve labeled true/false verdicts with explicit selection

doc/code/framework.md

@@ -283,6 +283,10 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 - Scorers with input limits use shared chunking to cover the scored content; each scorer owns context formatting, result aggregation, and uncertainty handling without changing the attack conversation.
 - A scorer is not limited to a message, it could be anything (e.g. was this tool called or was this file written). It receives a `Scorable`, which identifies that evidence, and an optional `ScoringExpectation`.
 - `TrueFalseScorer` and `FloatScaleScorer` define result families. `MessageScorer` adds message resolution and message-only policy on top of them.
+- `MultiLabelTrueFalseScorer` defines independent labeled boolean results. Its message
+  family aggregates pieces within each label. `TrueFalseScoreSelector` projects one
+  named verdict for attacks and evaluation while retaining the other labels as
+  intermediate scores. See [multiple labeled verdicts](./scoring/multi_label_true_false.md).
 - A scorer declares which evidence it reads, rather than the caller filtering evidence for it. A `MessageScorer` states the conversation roles and data types it reads on its `ScorerPromptValidator`.
 - Injected calls and results use simulated response roles and are excluded from message scoring
   unless explicitly selected. Prepended history is not an outbound request; execution scoring

doc/code/scoring/1_true_false_scorers.ipynb

@@ -729,6 +729,33 @@
     "  wrapped image `TrueFalseScorer` (True if *any* frame matches); an optional audio scorer is\n",
     "  AND-combined so both the visuals and the transcript must match."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "26",
+   "metadata": {},
+   "source": [
+    "## Selecting one label from a multi-label scorer\n",
+    "\n",
+    "`MultiLabelTrueFalseScorer` reports independent verdicts. When an attack or evaluator\n",
+    "expects one true/false result, use `TrueFalseScoreSelector` to choose a category\n",
+    "instead of combining unrelated labels.\n",
+    "\n",
+    "Given a configured `multi_label_scorer` that declares `harmful_response`:\n",
+    "\n",
+    "```python\n",
+    "from pyrit.score import TrueFalseScoreSelector\n",
+    "\n",
+    "objective_scorer = TrueFalseScoreSelector(\n",
+    "    scorer=multi_label_scorer,\n",
+    "    label=\"harmful_response\",\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "Use `objective_scorer` wherever a single `TrueFalseScorer` is expected. The label\n",
+    "must match one of the source scorer's `labels`; the other verdicts are retained\n",
+    "as intermediate scores."
+   ]
   }
  ],
  "metadata": {

doc/code/scoring/1_true_false_scorers.py

@@ -441,3 +441,25 @@
 # - **`VideoTrueFalseScorer`** — extracts frames from a `video_path` response and scores them with a
 #   wrapped image `TrueFalseScorer` (True if *any* frame matches); an optional audio scorer is
 #   AND-combined so both the visuals and the transcript must match.
+
+# %% [markdown]
+# ## Selecting one label from a multi-label scorer
+#
+# `MultiLabelTrueFalseScorer` reports independent verdicts. When an attack or evaluator
+# expects one true/false result, use `TrueFalseScoreSelector` to choose a category
+# instead of combining unrelated labels.
+#
+# Given a configured `multi_label_scorer` that declares `harmful_response`:
+#
+# ```python
+# from pyrit.score import TrueFalseScoreSelector
+#
+# objective_scorer = TrueFalseScoreSelector(
+#     scorer=multi_label_scorer,
+#     label="harmful_response",
+# )
+# ```
+#
+# Use `objective_scorer` wherever a single `TrueFalseScorer` is expected. The label
+# must match one of the source scorer's `labels`; the other verdicts are retained
+# as intermediate scores.

doc/code/scoring/multi_label_true_false.md

@@ -0,0 +1,113 @@
+# Multiple labeled true/false verdicts
+
+Some classifiers return several independent answers from one inference. For example,
+a response may refuse a harmful request without itself being harmful. Combining those
+answers with OR would lose that distinction.
+
+`MessageMultiLabelTrueFalseScorer` preserves one `Score` for each declared label in
+`score_category`. Each supported message piece produces a complete set of labeled
+verdicts. The message pipeline aggregates pieces within each label, using
+`TrueFalseScoreAggregator.OR` by default. Existing `MessageTrueFalseScorer` subclasses
+continue to return one aggregate verdict.
+
+## Implementing a classifier
+
+Declare nonempty, case-insensitively unique labels and implement
+`_score_piece_with_expectation_async`. Return a separate true/false `Score` with
+`score_category=[label]` for every declared label. Return `[]` when the classifier
+does not apply to the evidence. Missing, duplicate, or unknown labels are rejected
+before any scores are persisted.
+
+The following synthetic example reads classifier output from JSON. With PyRIT
+initialized, it can demonstrate the contract without calling a model:
+
+```python
+import json
+
+from pyrit.models import ContentScorable, MessagePiece, Score, ScoringExpectation
+from pyrit.score import (
+    MessageMultiLabelTrueFalseScorer,
+    ScorerPromptValidator,
+    TrueFalseScoreSelector,
+)
+
+
+class ExampleClassifier(MessageMultiLabelTrueFalseScorer):
+    def __init__(self) -> None:
+        super().__init__(
+            labels=["harmful_request", "response_refusal", "harmful_response"],
+            validator=ScorerPromptValidator(supported_data_types=["text"]),
+        )
+        self.calls = 0
+
+    async def _score_piece_with_expectation_async(
+        self, message_piece: MessagePiece, *, expectation: ScoringExpectation | None
+    ) -> list[Score]:
+        self.calls += 1
+        verdicts = json.loads(message_piece.converted_value)
+        return [
+            Score(
+                score_value=str(value).lower(),
+                score_type="true_false",
+                score_category=[label],
+                scorer_class_identifier=self.get_identifier(),
+                message_piece_id=message_piece.id,
+                scored_expectation=expectation,
+            )
+            for label, value in verdicts.items()
+        ]
+
+
+classifier = ExampleClassifier()
+evidence = ContentScorable(
+    value=json.dumps(
+        {
+            "harmful_request": True,
+            "response_refusal": True,
+            "harmful_response": False,
+        }
+    )
+)
+objective_scorer = TrueFalseScoreSelector(scorer=classifier, label="harmful_response")
+scores = await objective_scorer.score_async(scorable=evidence)
+print(classifier.calls, scores[0].get_value())
+# 1 False
+```
+
+A concrete target-backed scorer owns its `TargetJudge`, renders its own prompt,
+captures evidence with `_capture_judgment_evidence`, and passes the full expectation
+in `JudgmentRequest`. Its identifier must include the target and any additional
+behavioral configuration. The family base does not accept a `chat_target` parameter.
+`MultiLabelTrueFalseScorer` provides the same result contract for other evidence types.
+
+## Selecting, composing, and evaluating a verdict
+
+Use `TrueFalseScoreSelector` for an attack's objective/refusal scorer, boolean
+composition, or objective evaluation. The selector chooses by label, not output order,
+and its identity includes both the label and the source scorer. It preserves the
+source's target discovery and blocked-judge policy. Float thresholding is for float
+scores, not multi-label boolean results.
+
+One selector call returns one public projection and persists all labeled child
+verdicts as intermediate scores. The projection has its own score ID and selector
+identity. Default queries return public results. Query retained labels with
+`include_intermediate=True`:
+
+```python
+from pyrit.memory import CentralMemory
+
+memory = CentralMemory.get_memory_instance()
+request_scores = await memory.get_scores_async(
+    score_category="harmful_request",
+    include_intermediate=True,
+)
+```
+
+Calling the multi-label scorer directly returns and persists every label as a public
+result. Reading saved scores does not perform another inference. Separate scoring or
+selector calls can perform separate inferences; the selector does not cache them.
+
+Unsupported roles or modalities return `[]`. Unreadable or fully blocked evidence
+produces an undetermined score for each label. When `raise_if_scorer_blocks=False`,
+a blocked judge response does the same, preserving any retained error observation.
+The existing three-valued AND/OR aggregation applies independently to each label.

doc/myst.yml

@@ -163,6 +163,7 @@ project:
             - file: code/scoring/5_tool_call_scorer.ipynb
             - file: code/scoring/6_file_write_scorer.ipynb
             - file: code/scoring/7_program_execution_scorer.ipynb
+            - file: code/scoring/multi_label_true_false.md
         - file: code/memory/0_memory.md
           children:
             - file: code/memory/1_sqlite_memory.ipynb

pyrit/memory/memory_interface.py

@@ -3533,7 +3533,13 @@ def _execute_get_scores(
         if score_type:
             conditions.append(ScoreEntry.score_type == score_type)
         if score_category:
-            conditions.append(ScoreEntry.score_category == score_category)
+            conditions.append(
+                self._get_condition_json_array_match(
+                    json_column=ScoreEntry.score_category,
+                    property_path="$",
+                    array_to_match=[score_category],
+                )
+            )
         if sent_after:
             conditions.append(ScoreEntry.timestamp >= sent_after)
         if sent_before:

pyrit/score/__init__.py

@@ -100,6 +100,10 @@
     from pyrit.score.true_false.local_refusal_classifier_scorer import LocalRefusalClassifierScorer
     from pyrit.score.true_false.manual_scorer import ManualScorer
     from pyrit.score.true_false.message_tool_call_scorer import MessageToolCallScorer
+    from pyrit.score.true_false.multi_label_true_false_scorer import (
+        MessageMultiLabelTrueFalseScorer,
+        MultiLabelTrueFalseScorer,
+    )
     from pyrit.score.true_false.otel_tool_call_scorer import OtelToolCallScorer
     from pyrit.score.true_false.output_matches_scorer import OutputMatchesScorer
     from pyrit.score.true_false.program_execution_scorer import ProgramExecutionScorer
@@ -157,6 +161,7 @@
     from pyrit.score.true_false.true_false_fallback_scorer import TrueFalseFallbackScorer
     from pyrit.score.true_false.true_false_inverter_scorer import TrueFalseInverterScorer
     from pyrit.score.true_false.true_false_score_aggregator import TrueFalseAggregatorFunc, TrueFalseScoreAggregator
+    from pyrit.score.true_false.true_false_score_selector import TrueFalseScoreSelector
     from pyrit.score.true_false.true_false_scorer import MessageTrueFalseScorer, TrueFalseScorer
     from pyrit.score.true_false.video_true_false_scorer import VideoTrueFalseScorer
     from pyrit.score.true_false.wildguard_parser import WildGuardLabel, parse_wildguard_response
@@ -194,6 +199,9 @@
     "FloatScaleScorer": "pyrit.score.float_scale.float_scale_scorer",
     "MessageFloatScaleScorer": "pyrit.score.float_scale.float_scale_scorer",
     "MessageTrueFalseScorer": "pyrit.score.true_false.true_false_scorer",
+    "MultiLabelTrueFalseScorer": "pyrit.score.true_false.multi_label_true_false_scorer",
+    "MessageMultiLabelTrueFalseScorer": "pyrit.score.true_false.multi_label_true_false_scorer",
+    "TrueFalseScoreSelector": "pyrit.score.true_false.true_false_score_selector",
     "FloatScaleThresholdScorer": "pyrit.score.true_false.float_scale_threshold_scorer",
     "GarakExploitationDetector": "pyrit.score.true_false.garak_exploitation_scorer",
     "GarakExploitationScorer": "pyrit.score.true_false.garak_exploitation_scorer",

pyrit/score/scorer_evaluation/scorer_evaluator.py

@@ -30,6 +30,7 @@
     find_objective_metrics_by_eval_hash,
     replace_evaluation_results,
 )
+from pyrit.score.true_false.multi_label_true_false_scorer import MultiLabelTrueFalseScorer
 from pyrit.score.true_false.true_false_scorer import TrueFalseScorer
 
 if TYPE_CHECKING:
@@ -83,7 +84,12 @@ def __init__(self, scorer: Scorer) -> None:
 
         Args:
             scorer (Scorer): The scorer to evaluate.
+
+        Raises:
+            ValueError: If a multi-label scorer has no explicit single-label projection.
         """
+        if isinstance(scorer, MultiLabelTrueFalseScorer):
+            raise ValueError("Select a label with TrueFalseScoreSelector before evaluating a multi-label scorer.")
         self.scorer = scorer
 
     @classmethod

pyrit/score/true_false/float_scale_threshold_scorer.py

@@ -22,6 +22,7 @@
 from pyrit.score.observation.execution import _merge_observation_ids
 from pyrit.score.score_utils import ORIGINAL_FLOAT_VALUE_KEY
 from pyrit.score.scorer import Scorer
+from pyrit.score.true_false.multi_label_true_false_scorer import MultiLabelTrueFalseScorer
 from pyrit.score.true_false.true_false_scorer import TrueFalseScorer
 
 
@@ -67,6 +68,8 @@ def __init__(
         Raises:
             ValueError: If the threshold is non-finite or not in (0, 1].
         """
+        if isinstance(scorer, MultiLabelTrueFalseScorer):
+            raise ValueError("Multi-label true/false verdicts require TrueFalseScoreSelector, not a float threshold.")
         self._scorer = scorer
         self._threshold = threshold
         self._float_scale_aggregator = float_scale_aggregator