Skip to content

FEAT: Save techniques (part 14) - #4066

Open
varunj-msft wants to merge 531 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Techniques
Open

varunj-msft wants to merge 531 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Techniques

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 14 of saving Registry entries across restarts. Stacked on part 13 (#4052); part 20 (#4071) builds on it. This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Save-Converters-From-Registry...varunj-msft/13178-Save-Techniques

Design: 4.1 What gets saved, 4.3 What a recipe holds. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4052, which saves converters from the Registry. Until the earlier PRs merge, the diff here also shows their changes.

A technique made with New technique, or through POST /api/techniques, was gone after the backend restarted. It now saves by default and comes back at every start, like saved targets and converters.

  • POST /api/techniques takes "persist": true. A saved technique's name follows the saved-name rule (lowercase letters, digits and underscores, starting with a letter, up to 64 characters), or the request gets a 422. Its recipe is the create request without the name, plus who saved it.
  • A technique that uses a Temporary converter or adversarial target can't be saved, since those are gone after a restart (typed 400, /errors/not-saveable). Its attack type and attack parameters get the checks restore runs, so a type that can't be saved, a credential, or a Temporary entry named in the parameters is refused the same way. A name another technique has, or a tag that matches a technique's name or another spelling of a tag in use, is a 409 before anything is written, and storage that can't be written is a 503.
  • Restore rebuilds a saved technique with the call the API uses, after the converters and targets it names, and keeps who saved it out of the attack's settings. Techniques the startup scripts register are marked From configuration.
  • A scan's launch script that registers a technique under a saved technique's name or one of its tags stops the scan, even when the script catches the refusal, so the saved technique can't run in its place.
  • GET /api/techniques gives each technique an origin: saved, configuration or temporary.
  • The technique editor saves by default. When a converter or adversarial target it uses is Temporary, or the backend refuses those exact settings, the button reads Create as Temporary and a warning above the buttons gives the cause once. A taken name shows on the name field. A private technique can't be saved (422), and a saved one never records private in its recipe.
  • The Techniques table labels each technique Saved, From configuration or Temporary. The backend README and the GUI doc cover saving techniques.

What stays the same: a technique created without persist is Temporary, as before. Settings changed for one run in the Scanner (POST /api/techniques/{name}/edit) are never saved. Startup techniques work as before; they're only labeled From configuration.

Screenshots: New technique saves by default, the saved technique in the Techniques table, and a technique that uses a Temporary converter offered as Temporary with the reason.
techniques_1_new_technique_saves_by_default
techniques_2_saved_technique_row
techniques_3_temporary_converter_offers_temporary

Not covered:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A tag-related 409 conflict is stored only against the technique name, so correcting Tags leaves Save disabled and forces an unnecessary rename before retrying.

1 open issue
What changed in this PR

Extends registry persistence to attack techniques, building on #4052’s converter-saving work. Techniques can survive backend restarts while retaining the existing temporary-creation and per-run editing behavior.

Changes:

  • Adds saved technique requests, recipe metadata, restore integration, and dependency/conflict checks.
  • Makes the technique editor save by default, with Temporary fallback messaging and registry origin badges.
  • Includes stacked persistence safeguards, target ownership display, documentation, and isolated test storage.
File Description
tests/​unit/​setup/​test_targets_initializer.py Tests saved-target collision protection.
tests/​unit/​setup/​test_scorer_initializer.py Tests saved-scorer tagging safeguards.
tests/​unit/​setup/​test_converter_initializer.py Tests saved-converter replacement refusal.
tests/​unit/​setup/​test_configuration_loader.py Tests recipe storage configuration validation.
tests/​unit/​registry/​test_instance_registry.py Tests origin metadata and replacement safeguards.
tests/​unit/​registry/​test_converter_registry.py Tests saved-converter conflict recording.
tests/​unit/​registry/​test_attack_technique_registry.py Tests saved-technique registration safeguards.
tests/​unit/​backend/​test_technique_edit_service.py Updates creation calls to keyword arguments.
tests/​unit/​backend/​test_scorer_routes.py Tests scorer persistence and error responses.
tests/​unit/​backend/​test_scenario_suite_routes.py Tests sanitized storage failures.
tests/​unit/​backend/​test_scenario_suite_execution_service.py Tests saved-entry launch conflict reporting.
tests/​unit/​backend/​test_scenario_run_service.py Tests launch-script replacement safeguards.
tests/​unit/​backend/​test_scenario_preset_routes.py Tests sanitized storage responses and logs.
tests/​unit/​backend/​test_saved_entries.py Tests temporary dependency rejection.
tests/​unit/​backend/​test_mappers.py Tests target origin and ownership mapping.
tests/​unit/​backend/​test_initializer_service.py Tests unreadable initializer storage handling.
tests/​unit/​backend/​test_auth_middleware.py Tests authenticated object-ID lookup.
tests/​unit/​backend/​test_analytics_benchmark.py Tests benchmark recipe storage isolation.
tests/​unit/​backend/​conftest.py Isolates saved entries during backend tests.
tests/​end_to_end/​conftest.py Gives launched backends temporary recipe storage.
test_support/​analytics_benchmark.py Configures isolated benchmark recipe storage.
pyrit/​setup/​initializers/​tools.py Rejects saved-entry collisions during tool setup.
pyrit/​setup/​initializers/​targets.py Protects saved targets during automatic grouping.
pyrit/​setup/​initializers/​scorers.py Prevents saved scorers replacing configured defaults.
pyrit/​setup/​configuration_loader.py Adds recipe storage source configuration.
pyrit/​score/​true_false/​local_refusal_classifier_scorer.py Declares local classifier persistence restrictions.
pyrit/​score/​true_false/​audio_true_false_scorer.py Declares audio scorer persistence restrictions.
pyrit/​score/​scorer.py Adds scorer saveability metadata contracts.
pyrit/​score/​float_scale/​azure_content_filter_scorer.py Declares credential fallback behavior.
pyrit/​score/​float_scale/​audio_float_scale_scorer.py Declares audio scorer persistence restrictions.
pyrit/​registry/​instance_registry.py Tracks origins, owners, and saved-entry conflicts.
pyrit/​registry/​components/​scorer_registry.py Reserves the scorer types route name.
pyrit/​registry/​components/​attack_technique_registry.py Adds conflict prechecks and saved-technique safeguards.
pyrit/​prompt_target/​websocket_copilot_target.py Declares account-dependent persistence restriction.
pyrit/​prompt_target/​prompt_shield_target.py Declares credential and destination policies.
pyrit/​prompt_target/​openai/​openai_target.py Declares credential, header, and destination policies.
pyrit/​prompt_target/​litellm_chat_target.py Declares API-key fallback behavior.
pyrit/​prompt_target/​hugging_face/​hugging_face_chat_target.py Declares token and model persistence restrictions.
pyrit/​prompt_target/​http_target/​http_target.py Disallows saving credential-bearing raw requests.
pyrit/​prompt_target/​hack_a_prompt_target.py Declares session and cookie fallback behavior.
pyrit/​prompt_target/​github_copilot_target.py Declares token fallback behavior.
pyrit/​prompt_target/​common/​prompt_target.py Defines target persistence policy metadata.
pyrit/​prompt_target/​azure_ml_chat_target.py Declares credential and endpoint policies.
pyrit/​prompt_target/​azure_blob_storage_target.py Declares token and container destination policies.
pyrit/​prompt_target/​a2a_target.py Declares endpoint and agent-card destination policies.
pyrit/​models/​catalog/​target.py Adds target origin and ownership fields.
pyrit/​models/​catalog/​scorer.py Adds scorer origin field.
pyrit/​models/​catalog/​registry_recipe.py Defines saved recipes and restore reports.
pyrit/​models/​catalog/​__init__.py Exports recipe and restore models lazily.
pyrit/​converter/​converter.py Defines converter credential fallback metadata.
pyrit/​converter/​azure_speech_text_to_audio_converter.py Declares Speech credential fallback behavior.
pyrit/​converter/​azure_speech_audio_to_text_converter.py Declares Speech credential fallback behavior.
pyrit/​backend/​services/​technique_service.py Saves techniques and exposes their origins.
pyrit/​backend/​services/​scorer_service.py Saves scorers and exposes their origins.
pyrit/​backend/​services/​scenario_suite_execution_service.py Surfaces saved-entry launch conflict messages.
pyrit/​backend/​services/​scenario_run_service.py Detects saved-entry conflicts in launch scripts.
pyrit/​backend/​services/​runtime_lifecycle.py Restores saved entries and reports failures.
pyrit/​backend/​routes/​techniques.py Adds persistence context and typed save errors.
pyrit/​backend/​routes/​targets.py Adds persistence context and caller-specific ownership.
pyrit/​backend/​routes/​scorers.py Adds persistence context and save errors.
pyrit/​backend/​routes/​scenarios.py Documents saved-entry launch conflicts.
pyrit/​backend/​routes/​scenario_suites.py Sanitizes storage diagnostics and handles listing failures.
pyrit/​backend/​routes/​scenario_presets.py Sanitizes storage diagnostics and handles listing failures.
pyrit/​backend/​routes/​initializers.py Handles local storage failures with safe diagnostics.
pyrit/​backend/​routes/​converters.py Adds save errors and saved-removal protection.
pyrit/​backend/​routes/​common.py Shares owner extraction and save-refusal responses.
pyrit/​backend/​README.md Documents persistence APIs and origin semantics.
pyrit/​backend/​models/​techniques.py Adds persistence validation and origin metadata.
pyrit/​backend/​models/​targets.py Adds saveability metadata and persistence validation.
pyrit/​backend/​models/​scorers.py Adds scorer persistence validation.
pyrit/​backend/​models/​converters.py Adds saveability, origin, and persistence validation.
pyrit/​backend/​middleware/​auth.py Exposes the authenticated user’s object ID.
pyrit/​backend/​mappers/​technique_mappers.py Includes technique origin in responses.
pyrit/​backend/​mappers/​target_mappers.py Includes target origin and ownership display fields.
pyrit/​backend/​mappers/​converter_mappers.py Includes converter origin in responses.
frontend/​src/​utils/​registryOrigin.ts Defines shared origin labels.
frontend/​src/​utils/​registryName.ts Generates valid, available registry names.
frontend/​src/​utils/​registryName.test.ts Tests name validation and suggestions.
frontend/​src/​types/​index.ts Adds persistence, origin, and ownership types.
frontend/​src/​test-utils/​targetFixtures.ts Supports origin and ownership test data.
frontend/​src/​services/​errors.ts Recognizes typed save-refusal errors.
frontend/​src/​services/​errors.test.ts Tests save-refusal recognition.
frontend/​src/​components/​Techniques/​TechniqueEditor.tsx Saves by default and handles temporary dependencies.
frontend/​src/​components/​Techniques/​TechniqueEditor.test.tsx Updates nested converter save interactions.
frontend/​src/​components/​Registry/​TechniqueRegistry.tsx Displays technique origin badges.
frontend/​src/​components/​Registry/​ScorerRegistry.tsx Displays scorer origin badges.
frontend/​src/​components/​Registry/​ScorerRegistry.test.tsx Tests scorer origin labels.
frontend/​src/​components/​Registry/​OriginBadge.tsx Provides a shared origin badge.
frontend/​src/​components/​Registry/​ConverterRegistry.tsx Labels origins and disables saved-converter removal.
frontend/​src/​components/​Registry/​ConverterRegistry.test.tsx Tests origins and saved-removal restrictions.
frontend/​src/​components/​Converters/​SaveConverterDialog.tsx Clarifies that converter copies remain Temporary.
frontend/​src/​components/​Converters/​SaveConverterDialog.test.tsx Tests Temporary-copy messaging.
frontend/​src/​components/​Config/​TargetTable.tsx Shows origins, owners, and an ownership filter.
frontend/​src/​components/​Config/​TargetTable.test.tsx Tests origin labels and ownership filtering.
frontend/​src/​components/​Config/​TargetTable.styles.ts Styles ownership captions and filter controls.
frontend/​playwright.config.ts Isolates seeded backend recipe storage.
frontend/​e2e/​fixtures/​manual_send_backend.py Isolates offline backend recipe storage.
frontend/​e2e/​config.spec.ts Tests target saving and ownership display.
frontend/​e2e/​_targets.ts Adds origin and ownership mock fields.
doc/​gui/​0_gui.md Explains saving, Temporary entries, and removal limits.
doc/​getting_started/​pyrit_conf.md Documents recipe storage and restore failures.
.pyrit_conf_example Adds commented recipe storage configuration.

Review effort: Balanced

… a saveability answer once its request goes away

The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one
isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and
local refusal scorers. A saved entry still can't hold a typed key.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants