Skip to content

FEAT Configure scan targets before launch - #3794

Merged
Richard Lundeen (richlundeen) merged 12 commits into
microsoft:mainfrom
adimalkar:fix/3703-credential-free-scenario-catalog
Nov 10, 2026
Merged

Richard Lundeen (richlundeen) merged 12 commits into
microsoft:mainfrom
adimalkar:fix/3703-credential-free-scenario-catalog

Conversation

@adimalkar

@adimalkar Aditya Malkar (adimalkar) commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

Let users configure a scan before connecting a model. Browsing scenarios and selecting techniques no longer requires OPENAI_CHAT_MODEL or creates an OpenAI target behind the scenes.

  • Create or select the target under test, scorer, and any needed scoring/adversarial targets from the launch form. Creating or cancelling returns to the same draft.
  • Check each technique's requirements before work starts. Deterministic scorers need no scoring model; techniques without generation need no adversarial model. Missing or incompatible selections produce clear configuration errors.
  • Use explicit selections or configured defaults consistently across Python, CLI, API, and GUI. Environment-based initializers still work; one model serves multiple roles only when selected for each role.
  • Validate all suite children before launching any, and check saved component identities before resume so changed defaults cannot silently switch targets.
  • Update setup/scanner guidance and configuration/resiliency examples to explain role selection while retaining environment-based initialization.

Scanner with OPENAI_CHAT_MODEL unset

Both screenshots are real production builds with the variable unset and no configured targets or scorers. Before is the pre-change baseline (80f28af); after is this PR (99d7997). No scan or provider request was submitted.

Before After
The scanner is blocked by “Environment variable OPENAI_CHAT_MODEL is required.” The setup form opens: draft the objective and create/select targets and scorers. Launch stays disabled until required selections are supplied.
Before: scanner blocked because OPENAI_CHAT_MODEL is unset After: scanner setup opens with OPENAI_CHAT_MODEL unset and offers target and scorer creation

Credit

Fixes #3703. Thank you Aditya Malkar (@adimalkar). You found and pinned down this bug, and your original fix in this PR came first and was in good shape. I had made overlapping changes for the scan-target configuration work in #3926, so I pushed that branch here so both land as one change. Your diagnosis shaped the lazy default-scorer approach this uses.

Copilot AI balanced review requested due to automatic review settings November 1, 2026 17:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Decouple scenario discovery from runtime roles, validate selected dependencies before admission, and preserve explicit setup and saved identities across Python, API, CLI, GUI, and suites.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve explicit API connection-field validation in the imported HuggingFace event-loop regression by supplying its test-only token.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bind benchmark and converter dependencies, align effective factories and actual consumer capabilities, preserve scorer rubrics, and repair launch correction/confirmation UX. Cover frozen suite references at real registry admission.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain positive reuse and deliberate fresh edit-reruns, assert earlier drift rejection before datasets or new results, and configure the helper explicitly when constructing synthetic old saved runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Check every actual shared-helper consumer before work, including accepted simulated seeds with uses_adversarial=False. Preserve native helper success, objective-incompatible SKIP behavior and configured translation adaptation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Defer packaged compatibility reads until fixture execution. Compare complete target registry identity deltas and persisted offline run dependencies instead of prohibiting other specs' explicit OpenAI-shaped registrations.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep setup focused on selecting only needed scan roles and preserving environment-based initialization. Remove implementation and review-detail exposition.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve explicit scan roles through launch-form extraction and scenario run guards. Update readiness consumer mocks without changing production fallbacks or presentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Update paired documentation, suite contracts and readiness fixtures to the public import policy without behavior or notebook output changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assert disabled launch and actionable missing-target readiness before selecting the objective target. Preserve target-free estimates, direct API rejection and exact dataset execution checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@richlundeen
Richard Lundeen (richlundeen) force-pushed the fix/3703-credential-free-scenario-catalog branch from c798414 to 99d7997 Compare November 10, 2026 18:53
@richlundeen Richard Lundeen (richlundeen) changed the title FIX: list the scenario catalog without model credentials FEAT Configure scan targets before launch Nov 10, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 10, 2026
Merged via the queue into microsoft:main with commit 3cc6f65 Nov 10, 2026
61 checks passed
Julio César Suástegui (juliosuas) added a commit to juliosuas/PyRIT that referenced this pull request Nov 10, 2026
Resolve conflicts with microsoft#3794 (configure scan targets before launch):
- scenario_configuration_resolver: keep the exact-selection imports and
  the readiness/adversarial-override imports; resolve_configuration_async
  now accepts and forwards scoring_target_name so launch, estimate and
  dataset load keep the explicit scoring target with an exact selection.
- scenario_run_service: optional launch fields keep both
  scoring_target_name and example_selection.
- tests: patch resolve_target on the resolver class, as main's tests do,
  and cover scoring_target_name together with an exact selection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scenario discovery requires OpenAI credentials and one unavailable scenario blocks the entire catalog

3 participants