Skip to content

FEAT: Save converters from the Registry (part 13) - #4052

Open
varunj-msft wants to merge 485 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Converters-From-Registry
Open

varunj-msft wants to merge 485 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Converters-From-Registry

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 13 of saving Registry entries across restarts. Stacked on part 12 (#4051); part 14 (#4066) builds on it, and so does the side branch part 18 (#4069). This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Save-Converters-Scorers...varunj-msft/13178-Save-Converters-From-Registry

Design: 1. The contract. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4051, which saves converters and scorers created through the API. Until the earlier PRs merge, the diff here also shows their changes.

The New Converter dialog only made converters that are gone after the backend restarts, wherever it opens: the Converters tab, the converter panel in chat, and New converter in the technique editor. It now saves them by default, and the Converters and Scorers tables say which entries come back.

  • The dialog suggests a name from the type that no listed converter has, such as base64_converter. Keeping the _converter ending stays clear of the names the startup scripts register (base64, rot13), which would stop a saved converter of that name from restoring. The name field checks the backend's rule for saved names. A name the list already has, or one the backend says is taken (409), shows on the field, and focus moves there.
  • When a converter can't be saved, the button reads Create as Temporary and a warning says why: the type can't be saved, a file is uploaded, the chosen target is Temporary, or the backend refused those exact values (typed 400). Changing a refused value goes back to Save Converter. A file removed from a new converter is no longer sent. The warning sits above the buttons, outside the scrolling form, and gives the cause once. It uses role="status" so screen readers read it out, and target options mark Temporary targets.
  • GET /api/converters/types adds not_saveable_reason to each type, from the checks a save runs, including the one for types that need an uploaded file. The dialog keeps no copy of those rules.
  • The Converters table labels each converter Saved, From configuration or Temporary. A saved converter's Remove button is disabled, with a tooltip saying how to remove it for now. The Scorers table shows the same labels.
  • Save to registry on a converter's changed settings still makes a Temporary copy, and its dialog now says so.
  • The origin label moves from Config/TargetOriginBadge.tsx and utils/targetOrigin.ts to Registry/OriginBadge.tsx and utils/registryOrigin.ts. The target files change imports only.
  • The GUI doc covers saving converters and removing a saved one for now.

Screenshots: a new converter saves by default, a converter that uses a Temporary target is offered as Temporary with the reason, and the Converters table before and after a restart.
1_new_converter_saves_by_default
2_temporary_target_offers_temporary
4_converters_table_before_restart
5_converters_table_after_restart

Not covered:

Copilot AI balanced review requested due to automatic review settings November 7, 2026 23:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Caught duplicate-name collisions can let launch proceed with an unintended saved component, while synchronous restoration can block health and management requests.

2 open issues
What changed in this PR

This PR adds persistent registry entries to PyRIT’s converter-creation workflow and distinguishes saved entries from configuration-provided and temporary instances. It also includes supporting persistence, restoration, and target-management changes from the preceding work.

Changes:

  • Add saved-entry storage, validation, restoration, and protection against launch-script replacement.
  • Expose persistence and origin information through the API and registry UI.
  • Extend regression coverage, isolate test storage, and document saving and removal behavior.
File Description
tests/​unit/​setup/​test_targets_initializer.py Tests saved-target collision protection.
tests/​unit/​setup/​test_scorer_initializer.py Tests saved-scorer tagging protection.
tests/​unit/​setup/​test_converter_initializer.py Tests saved-converter replacement protection.
tests/​unit/​setup/​test_configuration_loader.py Tests recipe-storage configuration.
tests/​unit/​registry/​test_instance_registry.py Tests origins and recorded replacement refusals.
tests/​unit/​registry/​test_attack_technique_registry.py Tests saved-technique collision handling.
tests/​unit/​backend/​test_scorer_routes.py Tests scorer persistence and API errors.
tests/​unit/​backend/​test_scenario_suite_routes.py Tests sanitized storage failures.
tests/​unit/​backend/​test_scenario_suite_execution_service.py Tests saved-entry launch-conflict reporting.
tests/​unit/​backend/​test_scenario_run_service.py Tests launch-script replacement safeguards.
tests/​unit/​backend/​test_scenario_preset_routes.py Tests sanitized storage failures.
tests/​unit/​backend/​test_saved_entries.py Tests temporary-reference rejection.
tests/​unit/​backend/​test_mappers.py Tests target origin and ownership mapping.
tests/​unit/​backend/​test_initializer_service.py Tests local-storage failure responses.
tests/​unit/​backend/​test_auth_middleware.py Tests current-user ID lookup.
tests/​unit/​backend/​test_analytics_benchmark.py Tests benchmark recipe-storage isolation.
tests/​unit/​backend/​conftest.py Isolates saved entries for backend tests.
tests/​end_to_end/​conftest.py Gives test backends isolated recipe storage.
test_support/​analytics_benchmark.py Configures isolated benchmark recipe storage.
pyrit/​setup/​initializers/​tools.py Rejects saved-entry name collisions.
pyrit/​setup/​initializers/​targets.py Protects saved targets during grouping.
pyrit/​setup/​initializers/​scorers.py Prevents saved scorers replacing initializer defaults.
pyrit/​setup/​configuration_loader.py Adds recipe-storage configuration and validation.
pyrit/​score/​true_false/​local_refusal_classifier_scorer.py Declares model-related persistence restrictions.
pyrit/​score/​true_false/​audio_true_false_scorer.py Declares transcription-related persistence restrictions.
pyrit/​score/​scorer.py Adds scorer saveability metadata.
pyrit/​score/​float_scale/​azure_content_filter_scorer.py Declares server-credential fallback behavior.
pyrit/​score/​float_scale/​audio_float_scale_scorer.py Declares transcription-related persistence restrictions.
pyrit/​registry/​registry_recipe_storage.py Implements per-registry JSON recipe storage.
pyrit/​registry/​instance_registry.py Tracks origins, ownership, and replacement refusals.
pyrit/​registry/​components/​scorer_registry.py Reserves the scorer types route name.
pyrit/​registry/​components/​attack_technique_registry.py Supports saved metadata and collision protection.
pyrit/​prompt_target/​websocket_copilot_target.py Declares account-related persistence restrictions.
pyrit/​prompt_target/​prompt_shield_target.py Declares credential and destination rules.
pyrit/​prompt_target/​openai/​openai_target.py Declares credential, header, and destination rules.
pyrit/​prompt_target/​litellm_chat_target.py Declares API-key fallback behavior.
pyrit/​prompt_target/​hugging_face/​hugging_face_chat_target.py Declares token and model persistence restrictions.
pyrit/​prompt_target/​http_target/​http_target.py Disallows saving credential-bearing raw requests.
pyrit/​prompt_target/​hack_a_prompt_target.py Declares session and cookie fallback behavior.
pyrit/​prompt_target/​github_copilot_target.py Declares token fallback behavior.
pyrit/​prompt_target/​common/​prompt_target.py Defines target persistence-policy metadata.
pyrit/​prompt_target/​azure_ml_chat_target.py Declares credential and endpoint rules.
pyrit/​prompt_target/​azure_blob_storage_target.py Declares credential and container-address rules.
pyrit/​prompt_target/​a2a_target.py Declares endpoint and agent-card destination rules.
pyrit/​models/​catalog/​target.py Adds target origin and saver fields.
pyrit/​models/​catalog/​scorer.py Adds scorer origin.
pyrit/​models/​catalog/​registry_recipe.py Defines saved recipes and restoration reports.
pyrit/​models/​catalog/​__init__.py Lazily exports recipe models.
pyrit/​converter/​converter.py Adds credential-fallback metadata.
pyrit/​converter/​azure_speech_text_to_audio_converter.py Declares speech credential fallback.
pyrit/​converter/​azure_speech_audio_to_text_converter.py Declares speech credential fallback.
pyrit/​backend/​services/​scorer_service.py Saves scorers and reports their origins.
pyrit/​backend/​services/​scenario_suite_execution_service.py Exposes actionable saved-entry launch conflicts.
pyrit/​backend/​services/​scenario_run_service.py Detects launch-script replacement refusals.
pyrit/​backend/​services/​saved_entries.py Shares save validation and registration handling.
pyrit/​backend/​services/​runtime_lifecycle.py Restores saved entries and reports failures.
pyrit/​backend/​services/​converter_service.py Saves converters and exposes saveability rules.
pyrit/​backend/​routes/​targets.py Adds persistence errors and caller-aware responses.
pyrit/​backend/​routes/​scorers.py Handles scorer persistence requests and errors.
pyrit/​backend/​routes/​scenarios.py Documents saved-entry launch conflicts.
pyrit/​backend/​routes/​scenario_suites.py Sanitizes storage-error logging and responses.
pyrit/​backend/​routes/​scenario_presets.py Sanitizes storage-error logging and responses.
pyrit/​backend/​routes/​initializers.py Handles local-storage failures safely.
pyrit/​backend/​routes/​converters.py Handles persistence and saved-removal restrictions.
pyrit/​backend/​routes/​common.py Shares ownership and save-refusal response helpers.
pyrit/​backend/​README.md Documents persistence APIs and restrictions.
pyrit/​backend/​models/​targets.py Adds persistence validation and saveability fields.
pyrit/​backend/​models/​scorers.py Adds scorer persistence validation.
pyrit/​backend/​models/​converters.py Adds persistence, origin, and saveability fields.
pyrit/​backend/​middleware/​auth.py Exposes the authenticated user’s object ID.
pyrit/​backend/​mappers/​target_mappers.py Maps target origin and saver information.
pyrit/​backend/​mappers/​converter_mappers.py Maps converter origin.
frontend/​src/​utils/​registryOrigin.ts Defines shared origin labels.
frontend/​src/​utils/​registryName.ts Generates valid, unused registry names.
frontend/​src/​utils/​registryName.test.ts Tests name validation and suggestions.
frontend/​src/​types/​index.ts Adds persistence and origin API types.
frontend/​src/​test-utils/​targetFixtures.ts Supports origin and saver test data.
frontend/​src/​services/​errors.ts Recognizes typed save refusals.
frontend/​src/​services/​errors.test.ts Tests save-refusal classification.
frontend/​src/​components/​Techniques/​TechniqueEditor.test.tsx Updates converter save-button expectations.
frontend/​src/​components/​Registry/​ScorerRegistry.tsx Displays scorer origin badges.
frontend/​src/​components/​Registry/​ScorerRegistry.test.tsx Tests scorer origin labels.
frontend/​src/​components/​Registry/​OriginBadge.tsx Provides shared origin badges.
frontend/​src/​components/​Registry/​ConverterRegistry.tsx Shows origins and disables saved-entry removal.
frontend/​src/​components/​Registry/​ConverterRegistry.test.tsx Tests origins and saved-removal guidance.
frontend/​src/​components/​Converters/​SaveConverterDialog.tsx Clarifies that converter copies remain temporary.
frontend/​src/​components/​Converters/​SaveConverterDialog.test.tsx Tests temporary-copy wording.
frontend/​src/​components/​Config/​TargetTable.tsx Shows origins, savers, and an ownership filter.
frontend/​src/​components/​Config/​TargetTable.test.tsx Tests target origins and ownership filtering.
frontend/​src/​components/​Config/​TargetTable.styles.ts Styles saver text and visibility controls.
frontend/​playwright.config.ts Isolates seeded backend recipe storage.
frontend/​e2e/​fixtures/​manual_send_backend.py Isolates offline fixture recipe storage.
frontend/​e2e/​config.spec.ts Tests target saving and ownership presentation.
frontend/​e2e/​_targets.ts Adds origin and ownership mock fields.
doc/​gui/​0_gui.md Explains saved entries and manual removal.
doc/​getting_started/​pyrit_conf.md Documents recipe storage and restoration behavior.
.pyrit_conf_example Adds recipe-storage configuration guidance.

Review effort: Balanced

Comment thread pyrit/registry/instance_registry.py Resolved
@varunj-msft varunj-msft changed the title FEAT: Save converters from the Registry FEAT: Save converters from the Registry (part 13) Nov 9, 2026
varunj-msft and others added 21 commits November 10, 2026 22:33
# Conflicts:
#	tests/unit/registry/test_instance_registry.py
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	pyrit/backend/routes/converters.py
#	pyrit/backend/routes/targets.py
#	tests/unit/backend/test_runtime_lifecycle.py
Treating everything before the last "@" as a user name and password
rewrote a URL such as https://graph.example/users/alex@contoso.com/mail
into one that names the wrong host. Operators need the real URL (microsoft#4175:
don't redact what operators need), and the case it covered, an unescaped
"/", "?" or "#" in a password, is a malformed URL. The linear-time match,
which also finds a URL right after "_", stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	pyrit/registry/registry_recipe_restore.py
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	tests/unit/registry/test_registry_recipe_restore.py
# Conflicts:
#	pyrit/backend/services/scenario_suite_execution_service.py
# Conflicts:
#	pyrit/prompt_target/common/prompt_target.py
#	pyrit/prompt_target/openai/openai_target.py
# Conflicts:
#	pyrit/backend/README.md
#	pyrit/backend/mappers/target_mappers.py
#	pyrit/backend/models/targets.py
#	pyrit/backend/routes/targets.py
#	pyrit/backend/services/target_service.py
# Conflicts:
#	frontend/src/components/Config/CreateTargetDialog.tsx
#	frontend/src/types/index.ts
# Conflicts:
#	frontend/src/components/Config/TargetTable.styles.ts
#	frontend/src/components/Config/TargetTable.tsx
#	frontend/src/types/index.ts
#	pyrit/backend/mappers/target_mappers.py
#	tests/unit/backend/test_mappers.py
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one
isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and
local refusal scorers. A saved entry still can't hold a typed key.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants