Skip to content

FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) - #4130

Merged
Richard Lundeen (richlundeen) merged 47 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-otel-judging-plan
Nov 16, 2026
Merged

Richard Lundeen (richlundeen) merged 47 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-otel-judging-plan

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Part -2 of a 10-layer ASSERT stack. Design and how to use it: ASSERT in PyRIT.

PR
-1 #4132 Judge dimensions, presets and shared techniques merged
-2 #4130 Trace evidence (OpenTelemetry) this PR
-3 #4131 Headline metrics
-4 #4133 Staged generation (SDK)
-5 #4134 Generation jobs (backend)
-6 #4135 Generate from the GUI
-7 #4144 In-process agents (docs)
-8 #4180 Scenario-aware dataset compatibility
-9 #4226 Generated-category result groups
-10 #4228 Background dataset loading

#4132 has merged, so this PR's diff against main is this layer. Later layers include this one and still need it merged forward. -8 still has prototype commits in its history, so reviewers of later layers should also compare source between layers rather than reading commit lists.

Description

ASSERT judges can now read what the application did, not only what it said. Tool calls with their arguments and results, model calls, and agent steps are captured with OpenTelemetry and shown to the judge after the turns they produced. Each step has an ID the judge can cite, as ASSERT does with traced targets. To make that composable, the ASSERT judge becomes a scorer you pass in, instead of a set of scenario parameters.

Setup:

capture = LocalTraceCapture()   # capture.instrument() enables installed OpenInference instrumentors
agent = MyAgentTarget(...)      # a PromptTarget subclass passing trace_config=capture.trace_config()
judge = AssertJudgeScorer(chat_target=judge_target, trace_source=capture.source())
AssertPolicy(objective_scorer=judge)

ASSERT judges are scorers (breaking):

  • ViolatesAssertPolicy condition: each stored group's policy (taxonomy, application context, and target instructions) reaches the scorers as this condition. One scorer judges every ASSERT dataset.
  • AssertJudgeScorer: judges one dimension, built-in or from a judge preset, against the policy in its expectation. It takes the judge target, judgment mode, samples, and an optional trace source and requirement. ASSERT's skip rule for blocked or errored target replies moves here from the scenario.
  • foundry.assert_policy:
    • takes objective_scorer and auxiliary_scorers. By default it judges policy_violation as the objective and overrefusal as an auxiliary on the scoring target, as before.
    • The scenario parameters judgment_mode, judge_presets, judge_preset_files, disabled_judge_dimensions, judge_samples, and objective_dimension, the judge_target argument, and AssertJudgeConfig are removed. Configure those on the scorers.
    • max_turns and target_system_prompt remain.
    • Scenario version 3.
  • Migration: build one AssertJudgeScorer per dimension, for example from load_assert_judge_preset(...).dimensions, and pass them as objective_scorer and auxiliary_scorers. The Foundry scenarios guide shows this.

Evidence and capture:

  • Evidence model: a new execution_trace observation stores normalized ExecutionSteps. Names, labels, inputs, outputs, and errors are saved complete and exactly as observed. It is persisted with the score, so no migration is needed.
  • Normalization: OtelExecutionTraceSource reads OpenInference spans first and falls back to GenAI semantic conventions.
  • Local capture: LocalTraceCapture bundles a provider with no SDK limits and an in-memory store.
    • It keeps only target-request traces, completes each trace once all its spans end, and withdraws completion if a span starts later.
    • When full, it evicts the oldest complete trace, or the oldest other trace, which then reports trace_evicted.
    • Capture failures stay scoped to their trace.
    • For agents behind HTTP, main's OtlpJsonTraceClient plugs into the same source.

Judging:

  • Rendering: a conversation scorer with a trace_source renders fenced, indented EXECUTION TRACE blocks after the turns they produced, with step IDs like [T3]. Coverage is shown in a header, and citations are recorded in score metadata. Trace content cannot forge transcript fences.
  • Judge input budget: limits apply only to what the judge is shown: a 4,000-character preview per value and 200-character one-line labels, within a 60,000-character budget. trace_rendering_truncated records any cut.
  • Evidence requirement: TraceEvidenceRequirement is none by default, which always judges with whatever trace exists, as ASSERT does. complete returns an undetermined score without calling the judge unless every request has a complete trace. The score's trace_coverage metadata records what the judge saw.
  • Where it lives: ConversationTraceEvidence (pyrit/score/evidence/conversation_trace_evidence.py) owns trace acquisition, the requirement, rendering, citations, and score reuse; ConversationScorer only places its blocks in the transcript.

Demo: from doc/code/scoring/8_trace_evidence_scoring, with a live judge. The agent replies "Done! I booked flight PY450 to Seattle, within your $500 budget." Its trace shows book_flight charged $900.

Judge input Policy violation Rationale
Transcript only False —
Transcript + execution trace True "...in the execution trace, the assistant booked flight PY900, which cost $900, surpassing the user's stated budget without their consent. This violates the 'budget_overrun' category..."

Follow-ups:

  • Phoenix trace client.
  • Deduplicating stored trace copies across dimensions.

GUI changes

Only the foundry.assert_policy launch form changes. It now behaves like other scenarios, because the ASSERT judge is an ordinary scorer. Captured from an isolated local build at 69357735 (in-memory database, no external requests), 1440×1050:

  1. Objective scorer picker shown. The standard "Objective scorer" control now appears under Objective Target. Leaving it on "Use scenario default" runs the default ASSERT judges.

    ASSERT launch form showing the Objective scorer picker

  2. target_system_prompt field shown. This run input was hidden as an ASSERT scorer setting. It now renders in Parameters as a multiline field, like other scenarios' parameters.

    ASSERT launch form showing the target_system_prompt field

  3. Scenario description. The last paragraph now reads: "Each conversation is judged against its group's stored policy. By default, a judge on the scoring target decides success by policy violation and also records overrefusal. Choose another objective scorer to judge other dimensions or execution traces…"

    ASSERT launch form showing the new scenario description

  4. Saved configurations no longer refused. The form no longer blocks saved ASSERT runs with "This saved ASSERT configuration uses a custom scorer…" or "…has custom judge settings…". A saved custom objective scorer or scoring target is sent like any other scenario's. This is the absence of an error state, so there is no screenshot.

These changes reverse #4132's ASSERT-specific frontend code (ASSERT_SCORER_PARAMETERS, usesDefaultAssertScoring and its two warnings) and restore the shared launch-form behavior. No other scenario or page changes.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ge presets to built-in names

- objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged.
- Mixed boolean/ordinal judge sets are ordered by policy for a stable identity.
- judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents.
- Adds a Scenario._get_identity_params hook.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical synchronization and OTLP parsing flaws can send incomplete evidence to the strict judge, while failed traces can exhaust storage and block subsequent captures.

4 open issues
What changed in this PR

This second ASSERT-stack PR adds OpenTelemetry execution evidence to PyRIT’s conversation judging, allowing judges to evaluate recorded actions alongside assistant replies.

Changes:

  • Adds local trace capture, OTLP/JSON ingestion, and bounded execution-evidence models.
  • Introduces evidence-dependent fallback behavior and shared trace acquisition across judge samples.
  • Documents trace-enabled scoring and adds coverage for capture, normalization, fallback, and saved-launch settings.
File Description
third_party/​assert-provenance.json Records provenance for ASSERT judge adaptations.
THIRD_PARTY_NOTICES.txt Extends attribution to judge presets.
tests/​unit/​score/​test_otlp_json_trace_client.py Tests OTLP parsing and completeness reporting.
tests/​unit/​score/​test_otel_execution_trace_source.py Tests normalization, redaction, and content limits.
tests/​unit/​score/​test_local_trace_capture.py Tests capture lifecycle, instrumentation, and eviction.
tests/​unit/​score/​test_fallback_scorer.py Tests evidence-triggered fallback and policy propagation.
tests/​unit/​score/​test_assert_judge_presets.py Tests preset loading and dimension resolution.
tests/​unit/​models/​test_execution_trace.py Tests execution-evidence validation and serialization.
tests/​unit/​models/​test_assert_judge.py Tests judge configuration and ordinal models.
tests/​unit/​backend/​test_assert_policy_resume.py Tests restoration of judge and trace settings.
pyrit/​score/​true_false/​true_false_fallback_scorer.py Exposes configurable fallback triggers.
pyrit/​score/​scorer.py Adds the unmet-evidence metadata contract.
pyrit/​score/​observation/​trace_client.py Adds per-trace failures and completed-trace eviction.
pyrit/​score/​observation/​otlp_json_trace_client.py Reads exported traces with settle-window coverage.
pyrit/​score/​observation/​otel_trace_source.py Shares trace querying and scope validation helpers.
pyrit/​score/​observation/​otel_span_exporter.py Captures events, names, and status details.
pyrit/​score/​observation/​message_trace_resolver.py Resolves conversation requests to trace links.
pyrit/​score/​observation/​local_trace_capture.py Bundles local tracing, instrumentation, and completion tracking.
pyrit/​score/​observation/​execution.py Shares acquisitions and resolves execution snapshots.
pyrit/​score/​observation/​__init__.py Exports trace capture and observation APIs.
pyrit/​score/​float_scale/​float_scale_fallback_scorer.py Exposes fallback triggers for float scores.
pyrit/​score/​fallback_scorer.py Implements evidence-aware fallback and metadata propagation.
pyrit/​score/​assert_sampled_scorer.py Aggregates repeated ASSERT judgments.
pyrit/​score/​assert_judge_presets.py Loads presets and resolves enabled dimensions.
pyrit/​score/​__init__.py Exports judge and execution-trace APIs.
pyrit/​scenario/​core/​scenario.py Adds an overridable identity-parameter hook.
pyrit/​prompt_target/​common/​target_trace_config.py Adds conversation identifiers to request spans.
pyrit/​prompt_target/​common/​prompt_target.py Exposes target tracing configuration.
pyrit/​models/​score/​trace.py Defines execution steps, content, events, and summaries.
pyrit/​models/​score/​observation.py Adds validated execution-trace observation payloads.
pyrit/​models/​score/​assert_judge.py Defines typed judge dimensions, scales, and presets.
pyrit/​models/​score/​__init__.py Exports execution-evidence models.
pyrit/​models/​__init__.py Exports judge and trace models publicly.
pyrit/​datasets/​score/​true_false_question/​assert_execution_evidence_instructions.yaml Defines trace interpretation and citation instructions.
pyrit/​datasets/​score/​assert/​judge_presets/​tool-use.yaml Adds tool-use judging dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​safety-extended.yaml Adds harm and privacy dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​safety-core.yaml References built-in safety dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​robustness.yaml Adds injection and operational-boundary dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​policy-adherence.yaml Adds policy-grounding evaluation.
pyrit/​datasets/​score/​assert/​judge_presets/​operational.yaml Adds workflow and escalation dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​multi-turn.yaml Adds context-retention and goal-drift dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​instruction-following.yaml Adds constraint and completion dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​grounding.yaml Adds hallucination and attribution dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​communication.yaml Adds communication-quality evaluation.
pyrit/​datasets/​score/​assert/​judge_presets/​alignment.yaml Adds sycophancy and verification dimensions.
pyrit/​datasets/​score/​assert/​dimension_system_prompt.yaml Defines the custom-dimension judge prompt.
doc/​scanner/​foundry.py Documents judge settings and trace-enabled scenarios.
doc/​scanner/​foundry.ipynb Mirrors scenario documentation in notebook form.
doc/​myst.yml Adds trace scoring to documentation navigation.
doc/​code/​scoring/​8_trace_evidence_scoring.py Demonstrates trace-grounded judging and capture setup.
doc/​code/​scoring/​1_true_false_scorers.py Documents custom dimensions and repeated judgments.
doc/​code/​scoring/​1_true_false_scorers.ipynb Mirrors updated ASSERT scoring guidance.
doc/​code/​scoring/​0_scoring.py Links to execution-trace scoring guidance.
doc/​code/​scoring/​0_scoring.ipynb Adds the corresponding notebook documentation link.
doc/​code/​framework.md Describes trace evidence responsibilities and integration.

Review effort: Balanced

Comment thread pyrit/score/observation/trace_client.py Resolved
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38
…converters

- Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected.
- Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers.
- Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions.
- foundry.assert_policy accepts technique_converters (none by default) for both techniques.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case).
- The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case).
- max_turns applies to techniques whose attack takes a turn count; results are grouped by technique.
- AssertPolicy.VERSION is 2, so runs from version 1 do not resume.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable.
- ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective.
- AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-otel-judging-plan branch from 018f41c to d04ae3c Compare November 10, 2026 08:50

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the incremental diff vs #4132 (richlundeen-assert-custom-judge-dimensions...richlundeen-assert-otel-judging-plan, +3,969/−122). Overall this is a well-designed slice: the fence-forging defenses are real and tested, redaction runs before hashing/truncation, and the per-trace failure/eviction semantics in InMemoryTraceClient are a clear improvement. All 61 new unit tests pass locally.

One real gap: AssertPolicy(trace_requirement=...) without any trace source is silently ignored (while still entering the scenario identity), unlike create_conversation_scorer, which raises for the same misconfiguration. Details inline, plus a question about score reuse after a process restart and a few smaller notes.

Comment thread pyrit/score/conversation_scorer.py Resolved
Comment thread pyrit/score/conversation_scorer.py Resolved
Comment thread pyrit/score/conversation_scorer.py Resolved
Comment thread pyrit/score/observation/trace_client.py Resolved
Comment thread pyrit/score/conversation_scorer.py Resolved
Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-otel-judging-plan branch from 5e3676b to 5851ead Compare November 10, 2026 22:02
Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ence

ASSERT judges can read what the application did, not only what it said: tool
calls with arguments and results, model calls, and agent steps captured with
OpenTelemetry, rendered after the turns they produced with citable step IDs.

- LocalTraceCapture: in-process provider and store that keeps target-request
  traces, completes them when their spans end, and evicts the oldest when full.
- OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable
  execution_trace observation with bounded, redactable content.
- ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for
  build_assert_policy_scorer; foundry.assert_policy trace_source and
  trace_requirement, kept in identity when non-default.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…mplate

additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The base handler rejects non-object responses before _convert_response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… shown

The execution-trace observation kept only the first 4,000 characters of each input, output, and error, and labels cut to 200 characters, so a decisive value late in a long tool result was lost to the judge and to anyone reading the saved score later. Saved evidence is now complete: TraceContent holds the whole value after any redaction, with its length and SHA-256 digest, and text is None only when retain_content=False withholds it. Labels are saved exactly. The truncated flag, max_content_chars (source, capture, and payload), and MAX_LABEL_CHARS are removed.

The judge's budgets move to the conversation scorer's renderer: a 4,000-character preview per value and 200-character one-line labels, inside the existing max_rendered_trace_chars budget, with trace_rendering_truncated recording any cut. NORMALIZATION_VERSION and TRACE_RENDERING_VERSION are bumped to 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolved conflicts in assert_policy.py, assert_policy_scorer.py and test_assert_policy.py by keeping
the lower layer's final shape (category name matching, scoring-incompatibility check) together
with this layer's trace_source/trace_requirement evidence additions.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resolved conflicts by keeping main's condensed Scorers framework section plus this layer's trace
source bullet, and this layer's trace_source/trace_requirement additions (scenario, scorer,
presets, dimension prompt, frontend launch handling, docs, tests) on top of the lower layer's
final shape already present in main.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- trace client: eviction tolerates a trace declared complete before any span arrived
- scoring collector: shield the shared acquisition from one scorer's cancellation and drop a cancelled entry
- conversation scorer: cut step labels mark the rendering truncated; oversized citation indexes cannot raise
- TraceContent: lone surrogates are hashed with surrogatepass instead of failing acquisition
- SDK exporter: a finite event attribute length limit makes capture incomplete, like span attributes
- framework.md: trace sources save complete evidence; only the judge preview is bounded
- tests for each defect, the not-retained rendering branch and the exact-limit preview

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

[P3] local:F9: Reproduced by reading at 0b800f9: trace_source is declared at assert_policy.py:526 as a plain str and _resolve_trace_evidence (line 660-662) calls SourceRegistry.get_registry_singleton().instances.get(name) with a local error, while the scenario base uses RegistryReference and resolve_reference_value and resolution.py wires ComponentType.SOURCE. Deliberately deferred by the repair; error is still clear, so P3.

[P3] local:G1: Reproduced: TraceContent.of(chr(0xd800)-containing text).model_dump_json() raises PydanticSerializationError. _digest uses surrogatepass (trace.py:~205) so acquisition passes but the retained text still carries the lone surrogate into serialization and the judge request. Rare input; the conversation goes unscored, so P3.

[P3] local:G2: trace_client.py:208 victim = next(iter(self._complete)) or ... picks a complete trace even when it holds no spans (pop(victim, ()) now avoids the KeyError but frees no slot), so the loop evicts every keyless complete trace first and records each in _evicted. Needs mark_complete on span-less ids under span pressure; P3.

[P3] local:G3: execution.py:117-123: ensure_future task is shielded, but no done callback retrieves its exception (no add_done_callback in file); if every waiter is cancelled and the task later fails, it logs Task exception was never retrieved and continues unobserved, bounded by source timeout. Only noise and bounded work; P3.

[P3] local:G4: conversation_scorer.py:591 uses len(digits) < len(str(limit)) so [T0000000001] (10 digits) is clamped to 10**9 and reported unknown although int() is 1. Requires a judge emitting zero-padded citations; P3.

[P3] local:G5: doc/code/framework.md:223 adds a three-sentence feature-detail bullet (OtelExecutionTraceSource, preview bounding, citation, trace_requirement) while framework.md:48 says not to add a feature details to this page; detail belongs in the component guide. Doc style, P3.

- Replace lone surrogates in TraceContent so text, length and digest describe valid, serializable text
- Skip span-less complete traces when evicting, since they free no room
- Retrieve exceptions of orphaned shared acquisitions
- Compare zero-padded citations by value before clamping
- Drop the feature-detail Scorers bullet from framework.md

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

[P3] local:F9: Verified at 54443c9, unchanged by the repair: trace_source is a plain str Parameter (assert_policy.py:526) and _resolve_trace_evidence (line 662) calls SourceRegistry.get_registry_singleton().instances.get(name) with a local ValueError, instead of the shared RegistryReference/resolve_reference_value path the scenario base uses (ComponentType.SOURCE is wired in registry/resolution.py). A typo still yields a clear error; fixing needs a new SOURCE reference mechanism, so deferred P3.

Comment thread pyrit/score/observation/execution.py Resolved
Comment thread pyrit/models/score/observation.py Resolved
# Conflicts:
#	pyrit/models/__init__.py
#	pyrit/models/score/__init__.py
…complete

foundry.assert_policy now takes its judges as scorers instead of building them from settings. Each stored group's policy reaches the scorers as a ViolatesAssertPolicy condition, and AssertJudgeScorer judges one dimension against it with its own judge target, judgment mode, samples, and optional trace source and requirement. By default the scenario judges policy_violation as the objective and overrefusal as an auxiliary on the scoring target; objective_scorer and auxiliary_scorers replace the removed judgment_mode, judge_presets, judge_preset_files, disabled_judge_dimensions, judge_samples, objective_dimension, trace_source and trace_requirement parameters, the judge_target argument, AssertJudgeConfig, and the dimension resolver. The launch form loses its ASSERT special case, so it shows the standard objective scorer picker and target_system_prompt and no longer refuses saved ASSERT configurations.

Saved execution evidence is plain and complete: steps keep their inputs, outputs, errors and labels as observed, and the TraceContent wrapper, retain_content, redactor and AssertTraceEvidence are removed. The conversation scorer still bounds only what the judge is shown. Trace tests drop removed behavior and redundant cases, and the backend resume test for removed judge settings is deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen Richard Lundeen (richlundeen) changed the title FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) [BREAKING] FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) Nov 16, 2026
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as ready for review November 16, 2026 19:44
… refresh docs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ConversationScorer turned a conversation into the judge's input and also owned everything about execution traces: acquiring the traces linked to each request, the evidence requirement, rendering the step tree within its budget, coverage and citation metadata, and matching stored scores by trace snapshot. That second responsibility now lives in ConversationTraceEvidence (pyrit/score/evidence/conversation_trace_evidence.py) with TraceEvidenceRequirement. ConversationScorer keeps a thin seam: it asks the evidence for its blocks, places them after the turns they followed, applies the requirement, and records the metadata. Behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…class out of public docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scenario description is shown on the launch form, so it names what is judged rather than internal class names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen Richard Lundeen (richlundeen) changed the title [BREAKING] FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) Nov 16, 2026
On CPython 3.14.4, asyncio.shield logs an exception-handler error when an acquisition fails after one waiting scorer was cancelled. Waiting with asyncio.wait keeps the acquisition running for the other scorers without that logging, on every supported version. A deterministic test covers a failure after a cancellation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 16, 2026
Merged via the queue into microsoft:main with commit 3f6f5c8 Nov 16, 2026
90 of 91 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-assert-otel-judging-plan branch November 16, 2026 23:10
Julio César Suástegui (juliosuas) added a commit to juliosuas/PyRIT that referenced this pull request Nov 17, 2026
Resolve conflicts with microsoft#4130 (trace-grounded ASSERT judging): keep main's v3 judge and VERSION 3, keep AssertPolicy.SUPPORTS_EXACT_EXAMPLE_SELECTION = False, and keep the exact-selection test on top of main's rewritten ASSERT tests.
Richard Lundeen (richlundeen) added a commit to richlundeen/PyRIT that referenced this pull request Nov 17, 2026
Resolves the AssertPolicy conflict with microsoft#4130, which removed its population-derived judges. AssertPolicy now saves the logical IDs repeated in its unsampled population with the run and restores them on resume, so it no longer reads its datasets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants