Skip to content

MAINT: Simplify scenario resume around stored planned seed groups (seed 3) - #4366

Open
Richard Lundeen (richlundeen) wants to merge 6 commits into
microsoft:mainfrom
richlundeen:simplify-resume
Open

Richard Lundeen (richlundeen) wants to merge 6 commits into
microsoft:mainfrom
richlundeen:simplify-resume

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Previous: #4365 (seed 2). This PR builds on it; until #4350 and #4365 merge, review only the last commit, "MAINT: Simplify scenario resume around stored planned seed groups".

Description

Resume becomes one path: load the stored planned seed groups, restore any scenario state, and check the rebuilt plan against the stored one. #4350 stored each run's population but still re-ran overridden seed resolution on resume for its side effects, so those scenarios kept reading their datasets.

flowchart LR
  L["Load the stored<br/>planned seed groups"] --> S["_restore_resume_state_async<br/>(no-op by default)"]
  S --> A["Build atomic attacks"]
  A --> C{"Rebuilt plan equals<br/>stored plan?"}
  C -- yes --> R["Continue the run"]
  C -- no --> X["Refuse, naming the atomic<br/>groups that differ"]
Loading
  • Adds Scenario._restore_resume_state_async and stops calling _resolve_seed_groups_by_dataset_async on resume, removing the check for whether a scenario overrides it. Each scenario that keeps state from resolution restores it:

    Scenario Restored from
    LatentInjection the stored groups: sampling keeps every (family, trigger) pair its scorer needs
    PromptInject the goal scorer from parameters; technique templates saved in run metadata
    ApiKey the default scorer's excluded values, saved in run metadata
    PackageHallucination its datasets, since the package registries are too large to save
    AssertPolicy the logical IDs repeated in its unsampled population (used in unit names), saved in run metadata

    Every built-in scenario except PackageHallucination now resumes without reading datasets, so it also resumes after a dataset delete. Runs created before this PR fall back to reading the PromptInject, ApiKey, and AssertPolicy datasets.

  • Replaces _apply_persisted_run_plan's reorder-and-filter replay with _require_rebuilt_plan_matches, which requires the rebuilt plan to equal the stored plan and names the atomic groups missing, added, or changed.

  • Execute no longer records or checks a hash of its source conversation (PREPENDED_CONVERSATION_HASH_KEY is removed). A run whose source conversation was edited resumes on the history it planned instead of being refused.

  • Renames tests named for removed behavior and updates the resume section of the scenario instructions.

  • Merges main to resolve a conflict with FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) #4130's AssertPolicy refactor, which removed its population-derived judges.

Resuming a scenario after its dataset was deleted reported "no seeds found
in memory. Call prepare_async() ...", which resume never does. After an edit
or refresh it reported only "missing N planned seed group(s)".

Resume now names the deleted datasets, or the resolved datasets whose content
changed, and ends with the two recoveries: re-import the content the run used,
or drop scenario_result_id to start a new run. Only a dataset that is actually
absent is reported as deleted; other resolution errors propagate unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new run now stores the population it was built from, in the same
transaction as its ScenarioResult, in ScenarioPlanSeedGroupEntries. Resume
rebuilds from those groups and never prepares, resolves, or resamples the
population datasets, so editing, refreshing, or deleting a dataset no longer
breaks resume. Scenarios that override seed resolution still run it on
resume for the state it sets up, and discard its groups.

This also keeps exactly the copies of duplicate groups the first run
sampled (microsoft#4157), and removes the legacy replay: objective_hashes,
_apply_persisted_objectives, and rebuilding plans for results without one.
Runs created before this change can no longer be resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… seeds

Resume now receives exactly the planned population, in order and with its
duplicate copies, so AdaptiveScenario builds and names its atomics the same
way a new run does and only reads its saved technique choices from the plan.
This removes _restore_planned_populations and the separate resume branches
for the baseline and per-dataset atomics.

AtomicAttack.keep_seed_groups_with_hashes only served the removed
objective_hashes replay, so it is deprecated for removal in 1.4.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume rebuilds a run from its stored planned seed groups, but dataset media
cleanup didn't count those groups as references, so deleting a dataset could
remove a file a paused run still needed. stored_media_keys now also reads
media paths from planned seed groups of runs that haven't completed, using
the has_media index. Planned seed group writes go through the same dataset
reference guard as seeds and messages, so a new run can't plan media that is
already queued for deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume no longer re-runs a scenario's seed resolution for its side effects.
Scenarios that keep state from resolution restore it in a new
_restore_resume_state_async hook: LatentInjection rebuilds its scorer from
the stored groups, PromptInject and ApiKey restore templates and excluded
values saved in run metadata, and PackageHallucination and AssertPolicy
still read their datasets. Scenarios that only reshape groups no longer
read datasets on resume, so they resume after a dataset delete too.

Resume now requires the rebuilt plan to equal the stored plan and names
the atomic groups that differ, replacing the reorder-and-filter replay.
Execute no longer records or checks a hash of its source conversation,
since the stored group holds the history it planned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Resume validation rejects unchanged, partially cached AdversarialBenchmark runs because the rebuilt atomic ordering differs from the saved plan; restore the saved ordering and add a regression test.

1 open issue
What changed in this PR

Building on stored populations and media retention in #4350 and #4365, this PR simplifies PyRIT scenario resume so most scenarios no longer reread their datasets.

Changes:

  • Unifies resume around stored seed groups, explicit state restoration, and exact plan validation.
  • Restores scorer and template state from saved data where possible, retaining necessary dataset reads for AssertPolicy and PackageHallucination.
  • Removes Execute’s source-history hash check and updates regression tests and scenario guidance.
File Description
tests/​unit/​scenario/​scenarios/​technique/​test_execute.py Verifies resume preserves planned history after source edits.
tests/​unit/​scenario/​scenarios/​adaptive/​test_text_adaptive.py Updates stored-population and saved-choice expectations.
tests/​unit/​scenario/​garak/​test_prompt_inject.py Tests template restoration after dataset deletion.
tests/​unit/​scenario/​garak/​test_latent_injection.py Tests saved cases and scorer reconstruction.
tests/​unit/​scenario/​garak/​test_exploitation.py Updates the plan-mismatch error expectation.
tests/​unit/​scenario/​garak/​test_api_key.py Tests scorer restoration without corpus reads.
tests/​unit/​scenario/​core/​test_scenario.py Covers stored populations, restoration hooks, and resume failures.
tests/​unit/​scenario/​core/​test_scenario_retry.py Removes obsolete objective-filter replay tests.
tests/​unit/​scenario/​core/​test_scenario_dataset_overrides.py Updates resume comments and configuration-mismatch expectations.
tests/​unit/​scenario/​core/​test_atomic_attack.py Checks deprecated filtering emits warnings.
tests/​unit/​memory/​test_scenario_plan_seed_groups_migration.py Covers migration round trips and SQL Server compilation.
tests/​unit/​memory/​test_planned_seed_groups.py Tests ordered storage, duplicates, and media references.
tests/​unit/​memory/​test_dataset_lifecycle.py Tests planned-media retention and deletion guards.
tests/​unit/​memory/​test_dataset_lifecycle_migration.py Includes planned seed storage in schema expectations.
tests/​unit/​backend/​test_scenario_resume.py Updates persistence interception for planned-result writes.
pyrit/​scenario/​scenarios/​technique/​execute.py Removes source-conversation hash storage and validation.
pyrit/​scenario/​scenarios/​garak/​prompt_inject.py Saves templates and restores goal-scoring state.
pyrit/​scenario/​scenarios/​garak/​package_hallucination.py Restores scorer state from package registries.
pyrit/​scenario/​scenarios/​garak/​latent_injection.py Rebuilds scorer coverage from stored groups.
pyrit/​scenario/​scenarios/​garak/​api_key.py Saves and restores default scorer exclusions.
pyrit/​scenario/​scenarios/​foundry/​assert_policy.py Restores judges and repeated IDs from datasets.
pyrit/​scenario/​scenarios/​adaptive/​adaptive_scenario.py Rebuilds attacks from stored populations and choices.
pyrit/​scenario/​core/​scenario.py Adds stored-population loading, restoration hooks, and exact validation.
pyrit/​scenario/​core/​atomic_attack.py Deprecates the former resume filtering helper.
pyrit/​memory/​memory_session.py Guards planned-group writes against deleted media.
pyrit/​memory/​memory_models.py Defines stored planned groups and media extraction.
pyrit/​memory/​memory_interface.py Persists results with populations and reads them back.
pyrit/​memory/​dataset_lifecycle.py Retains media referenced by incomplete runs.
pyrit/​memory/​alembic/​versions/​c7d1e9a3b5f2_add_scenario_plan_seed_groups.py Creates indexed storage for planned seed groups.
.github/​instructions/​scenarios.instructions.md Documents state restoration and exact resume validation.

Review effort: Balanced

Comment on lines +1868 to +1869
rebuilt = self._build_run_plan()
if rebuilt != stored_plan:
Resolves the AssertPolicy conflict with microsoft#4130, which removed its population-derived judges. AssertPolicy now saves the logical IDs repeated in its unsampled population with the run and restores them on resume, so it no longer reads its datasets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants