5 - FEAT: Latest-response Scoring by richlundeen · Pull Request #3644 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-10-29T14:52:33Z

Applies the selected scorer to the latest response in the open conversation from the scorer pane. Reuses valid existing results without changing human scores or future-response scoring.

Changes:

Add Apply scorer inside the scorer pane after selection; hide it for Off.
Score only the latest saved assistant response in the open conversation.
Keep a complete matching result without a new evaluation; preserve human scores and future-response scoring.
Show progress, the result and rationale, and an Already scored notice; remove the all-conversations scoring text.

Layer 5/10 | Previous: #3647 | Final: #3648 | Blocked by: #3647

Diff is cumulative until preceding layers merge.

Before (original GUI)

The original scorer pane showed a future-responses explanation and had no Apply action for the latest saved response.

After

Apply scorer is inside the scorer pane after selection. The latest response's result and rationale are shown; earlier responses and other conversations are unchanged.

Applying the same scorer again keeps the existing valid result and rationale. Already scored confirms that no new evaluation ran.

Video

latest-response-scoring-production.webm

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Move history into feature routes, add stored ID lookup, and retain the current chat draft. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve discard blocking before resetting the composer and use shared encoded continuation and scenario routes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep an explicit not-found page and Home recovery link at unknown URLs. Preserve the last valid chat and dataset views without changing legacy aliases or Registry index behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Prefer the existing retained draft when entering Chat from another view. Keep an already open chat in place so live work is not replaced by an older snapshot. Cover the workflow with the real History table, Chat tabs, composer and reset controls. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Move score matching into Scorer.find_matching_scores so reuse compares the expectation score_async prepares, share chat evaluation and bounded publication between Send and Apply, project the latest-response result, outcome, and canonical response ID from the backend, and keep Apply status in ChatWindow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Use shared readiness and completion watchdogs instead of a short scheduling deadline while preserving the concurrent settings edit assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Behnam (behnam-o)

[GHCP]: Verdict: APPROVE.

PR 5/10 - Latest-response Scoring. Please address and resolve all 3 numbered inline comments in this review. Approval does not waive any comment.

[GHCP]: MEDIUM - Conversation-wrapper scorers cannot match their persisted results.
[GHCP]: MEDIUM - Apply creates judge conversations without attack ownership.
[GHCP]: MEDIUM - Failed refresh shows an old verdict against a new objective.

Scope: the feature increment 287e94a10c1b0f84e82056ef0cccd37e842de767 -> 3ecc1bfd66979b09526acf4f2af2498ef4c0424c. Static review only; tests, builds, and browser validation were not run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep the messages GET aligned with current attack score, response, outcome, and failure state across successful and failed sends. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen)

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Persist private participation provenance and validate every participating conversation snapshot before reusing aggregate results. Preserve original same-input anchors for inverter, threshold and fallback roots. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Treat private JSON scalar strings as unknown provenance rather than decoding them into trusted child participation. Cover real persisted string/dict/NULL projection and stale snapshot reuse. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrate exact upstream main commit 9cd17cf and preserve its evaluator default regression coverage and refreshed metrics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrate exact upstream a214848 SQL/XSS scorer, regression and synchronized documentation changes without altering prior scorer freshness repairs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Incorporate exact main 851d40f without changing scorer sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Invalidate cached count-only default scores when detection patterns change while retaining custom and exclusion identity behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrate main 0ffa2fc, preserving legacy pruned-conversation history lookup and current chat navigation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve both integration branches and remove the obsolete App mock binding exposed by merging. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Retain independent registry identity rows, the supported adversarial default, and removal of objective defaults. Remove the duplicate identity documentation paragraph introduced by the merge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve atomic stored-message metadata, shared owned-write cancellation, and chat scoring policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve manual chat scoring and content-sensitive regex identities while integrating typed scorer hooks and dependency URL-decoding policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep equality checks for pattern-only scorers while recognizing default-only preprocessing in merged shell scoring. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Re-point d43b1bcf9c56 after e7c4a2d9f013 from microsoft#3644, so the chain keeps a single head. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FEAT: Add expandable sidebar navigation

dcf012f

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Oct 29, 2026

6 - FEAT: Scan Results and Live Chats #3643

Merged

Richard Lundeen (richlundeen) and others added 5 commits October 29, 2026 09:36

FIX: Preserve sidebar tooltips and share responsive queries

4a72b94

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FEAT: Consolidate attack and scanner history

ca81736

Move history into feature routes, add stored ID lookup, and retain the current chat draft. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Keep editor drafts until new-attack navigation commits

22faf6b

Preserve discard blocking before resetting the composer and use shared encoded continuation and scenario routes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

hannahwestra25 assigned Behnam (behnam-o) Oct 29, 2026

Richard Lundeen (richlundeen) and others added 11 commits October 29, 2026 11:40

FEAT: Normalize registry identities

2b4b45d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Align registry browser checks and identity docs

9fc4c51

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FEAT: Reuse last-used chat and scan selections

65fe249

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Preserve scorer preferences and repair target setup coverage

c548838

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Preserve explicit unavailable scorer overrides

0ea5a7a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: Clarify the current adversarial fallback order

4428222

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Remove obsolete effect lint suppression

287e94a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FEAT: Latest-response Scoring

10c7fe8

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Scope Apply status to its scorer and objective

dbf8cf4

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Apply layer-4 scorer preference guards to Apply

3ecc1bf

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) force-pushed the richlundeen-chat-scoring branch from 8129d80 to 3ecc1bf Compare October 29, 2026 19:13

Merge origin/main and stabilize chat scoring concurrency regression

7f925d7

Use shared readiness and completion watchdogs instead of a short scheduling deadline while preserving the concurrent settings edit assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Behnam (behnam-o) approved these changes Oct 30, 2026

View reviewed changes

Comment thread pyrit/backend/services/chat_scoring_service.py Resolved

Comment thread pyrit/backend/services/chat_scoring_service.py Resolved

Comment thread frontend/src/components/Chat/ChatWindow.tsx Resolved

Richard Lundeen (richlundeen) and others added 8 commits October 31, 2026 15:14

Merge origin/main into PR 3650

7279872

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge PyRIT 3642 dependency into PR 3650

050678a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge latest main before PR repair

6aa0feb

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

fix(frontend): repair consolidated history feedback

aa611d7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge main into PR 3642 repair baseline

3b01664

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Restore focused chat scoring test waits

1ac07c1

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge exact origin/main into PR 3650 preserving consolidated navigation

a9f4e93

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge prepared PyRIT PR 3642 dependency into PR 3650

9f85042

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Include authoritative scoring projection in chat E2E mock

9304421

Keep the messages GET aligned with current attack score, response, outcome, and failure state across successful and failed sends. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) commented Nov 1, 2026

View reviewed changes

Merge latest main into PR 3644

b1bc336

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 2, 2026 15:55

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 2, 2026 16:18

View session

Copilot AI reviewed Nov 2, 2026

Richard Lundeen (richlundeen) and others added 6 commits November 2, 2026 08:40

Merge latest main scorer evaluation defaults

75e232e

Integrate exact upstream main commit 9cd17cf and preserve its evaluator default regression coverage and refreshed metrics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge main output scorer detection improvements

669b95f

Integrate exact upstream a214848 SQL/XSS scorer, regression and synchronized documentation changes without altering prior scorer freshness repairs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge main dependency override update into chat scoring

606170e

Incorporate exact main 851d40f without changing scorer sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Include default regex content in scorer identities

b1ac022

Invalidate cached count-only default scores when detection patterns change while retaining custom and exclusion identity behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into richlundeen-converter-registry

965afdc

Richard Lundeen (richlundeen) marked this pull request as ready for review November 3, 2026 01:40

Richard Lundeen (richlundeen) and others added 10 commits November 2, 2026 17:59

Merge selected main into PR microsoft#3651

6a1f5e8

Integrate main 0ffa2fc, preserving legacy pruned-conversation history lookup and current chat navigation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge dependency PR microsoft#3651 into PR microsoft#3647

e4e63c6

Preserve both integration branches and remove the obsolete App mock binding exposed by merging. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve dispatch evidence and refresh stale send metadata

58b4c8c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Cover retained invocation evidence in route contracts

b0b5b10

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge dependency PR microsoft#3647 into PR microsoft#3644

e05cc58

Preserve atomic stored-message metadata, shared owned-write cancellation, and chat scoring policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge selected main into PR microsoft#3644

684710c

Preserve manual chat scoring and content-sensitive regex identities while integrating typed scorer hooks and dependency URL-decoding policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Include SQL quote filtering in scorer identity

498230a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve first-send evidence across created-attack route commits

5a3e201

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

TST: Respect default-only regex scoring policies in identity assertions

de33e67

Keep equality checks for pattern-only scorers while recognizing default-only preprocessing in merged shell scoring. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 3, 2026

Merged via the queue into microsoft:main with commit 335f989 Nov 3, 2026
61 of 64 checks passed

Richard Lundeen (richlundeen) deleted the richlundeen-chat-scoring branch November 3, 2026 10:07

Richard Lundeen (richlundeen) mentioned this pull request Nov 3, 2026

10 - FEAT: Media Drag and Drop #3648

Merged

Navigation Menu

Repository navigation

5 - FEAT: Latest-response Scoring - #3644

5 - FEAT: Latest-response Scoring#3644
Richard Lundeen (richlundeen) merged 59 commits into
microsoft:mainfrom
richlundeen:richlundeen-chat-scoring

Richard Lundeen (richlundeen) commented Oct 29, 2026 •

edited

Loading

Uh oh!

Behnam (behnam-o) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Richard Lundeen (richlundeen) left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Richard Lundeen (richlundeen) commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before (original GUI)

After

Video

Uh oh!

Behnam (behnam-o) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Richard Lundeen (richlundeen) left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Oct 29, 2026 •

edited

Loading