Repository navigation
FEAT: Save converters from the Registry (part 13) - #4052
Open
varunj-msft wants to merge 485 commits into
Open
varunj-msft wants to merge 485 commits into
varunj-msft wants to merge 485 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Caught duplicate-name collisions can let launch proceed with an unintended saved component, while synchronous restoration can block health and management requests.
2 open issues
What changed in this PR
This PR adds persistent registry entries to PyRIT’s converter-creation workflow and distinguishes saved entries from configuration-provided and temporary instances. It also includes supporting persistence, restoration, and target-management changes from the preceding work.
Changes:
- Add saved-entry storage, validation, restoration, and protection against launch-script replacement.
- Expose persistence and origin information through the API and registry UI.
- Extend regression coverage, isolate test storage, and document saving and removal behavior.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging protection. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement protection. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe-storage configuration. |
| tests/unit/registry/test_instance_registry.py | Tests origins and recorded replacement refusals. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique collision handling. |
| tests/unit/backend/test_scorer_routes.py | Tests scorer persistence and API errors. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch-conflict reporting. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_saved_entries.py | Tests temporary-reference rejection. |
| tests/unit/backend/test_mappers.py | Tests target origin and ownership mapping. |
| tests/unit/backend/test_initializer_service.py | Tests local-storage failure responses. |
| tests/unit/backend/test_auth_middleware.py | Tests current-user ID lookup. |
| tests/unit/backend/test_analytics_benchmark.py | Tests benchmark recipe-storage isolation. |
| tests/unit/backend/conftest.py | Isolates saved entries for backend tests. |
| tests/end_to_end/conftest.py | Gives test backends isolated recipe storage. |
| test_support/analytics_benchmark.py | Configures isolated benchmark recipe storage. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry name collisions. |
| pyrit/setup/initializers/targets.py | Protects saved targets during grouping. |
| pyrit/setup/initializers/scorers.py | Prevents saved scorers replacing initializer defaults. |
| pyrit/setup/configuration_loader.py | Adds recipe-storage configuration and validation. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares model-related persistence restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares transcription-related persistence restrictions. |
| pyrit/score/scorer.py | Adds scorer saveability metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares server-credential fallback behavior. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares transcription-related persistence restrictions. |
| pyrit/registry/registry_recipe_storage.py | Implements per-registry JSON recipe storage. |
| pyrit/registry/instance_registry.py | Tracks origins, ownership, and replacement refusals. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/attack_technique_registry.py | Supports saved metadata and collision protection. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares account-related persistence restrictions. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination rules. |
| pyrit/prompt_target/openai/openai_target.py | Declares credential, header, and destination rules. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback behavior. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model persistence restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Disallows saving credential-bearing raw requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback behavior. |
| pyrit/prompt_target/github_copilot_target.py | Declares token fallback behavior. |
| pyrit/prompt_target/common/prompt_target.py | Defines target persistence-policy metadata. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and endpoint rules. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares credential and container-address rules. |
| pyrit/prompt_target/a2a_target.py | Declares endpoint and agent-card destination rules. |
| pyrit/models/catalog/target.py | Adds target origin and saver fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin. |
| pyrit/models/catalog/registry_recipe.py | Defines saved recipes and restoration reports. |
| pyrit/models/catalog/__init__.py | Lazily exports recipe models. |
| pyrit/converter/converter.py | Adds credential-fallback metadata. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Declares speech credential fallback. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Declares speech credential fallback. |
| pyrit/backend/services/scorer_service.py | Saves scorers and reports their origins. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes actionable saved-entry launch conflicts. |
| pyrit/backend/services/scenario_run_service.py | Detects launch-script replacement refusals. |
| pyrit/backend/services/saved_entries.py | Shares save validation and registration handling. |
| pyrit/backend/services/runtime_lifecycle.py | Restores saved entries and reports failures. |
| pyrit/backend/services/converter_service.py | Saves converters and exposes saveability rules. |
| pyrit/backend/routes/targets.py | Adds persistence errors and caller-aware responses. |
| pyrit/backend/routes/scorers.py | Handles scorer persistence requests and errors. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage-error logging and responses. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage-error logging and responses. |
| pyrit/backend/routes/initializers.py | Handles local-storage failures safely. |
| pyrit/backend/routes/converters.py | Handles persistence and saved-removal restrictions. |
| pyrit/backend/routes/common.py | Shares ownership and save-refusal response helpers. |
| pyrit/backend/README.md | Documents persistence APIs and restrictions. |
| pyrit/backend/models/targets.py | Adds persistence validation and saveability fields. |
| pyrit/backend/models/scorers.py | Adds scorer persistence validation. |
| pyrit/backend/models/converters.py | Adds persistence, origin, and saveability fields. |
| pyrit/backend/middleware/auth.py | Exposes the authenticated user’s object ID. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and saver information. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin. |
| frontend/src/utils/registryOrigin.ts | Defines shared origin labels. |
| frontend/src/utils/registryName.ts | Generates valid, unused registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds persistence and origin API types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and saver test data. |
| frontend/src/services/errors.ts | Recognizes typed save refusals. |
| frontend/src/services/errors.test.ts | Tests save-refusal classification. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates converter save-button expectations. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Displays scorer origin badges. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests scorer origin labels. |
| frontend/src/components/Registry/OriginBadge.tsx | Provides shared origin badges. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Shows origins and disables saved-entry removal. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests origins and saved-removal guidance. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Clarifies that converter copies remain temporary. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests temporary-copy wording. |
| frontend/src/components/Config/TargetTable.tsx | Shows origins, savers, and an ownership filter. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests target origins and ownership filtering. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles saver text and visibility controls. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline fixture recipe storage. |
| frontend/e2e/config.spec.ts | Tests target saving and ownership presentation. |
| frontend/e2e/_targets.ts | Adds origin and ownership mock fields. |
| doc/gui/0_gui.md | Explains saved entries and manual removal. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restoration behavior. |
| .pyrit_conf_example | Adds recipe-storage configuration guidance. |
Review effort: Balanced
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 02:55
This was referenced Nov 8, 2026
# Conflicts: # tests/unit/registry/test_instance_registry.py # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # pyrit/backend/routes/converters.py # pyrit/backend/routes/targets.py # tests/unit/backend/test_runtime_lifecycle.py
Treating everything before the last "@" as a user name and password rewrote a URL such as https://graph.example/users/alex@contoso.com/mail into one that names the wrong host. Operators need the real URL (microsoft#4175: don't redact what operators need), and the case it covered, an unescaped "/", "?" or "#" in a password, is a malformed URL. The linear-time match, which also finds a URL right after "_", stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # pyrit/registry/registry_recipe_restore.py # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # pyrit/backend/services/scenario_suite_execution_service.py
# Conflicts: # pyrit/prompt_target/common/prompt_target.py # pyrit/prompt_target/openai/openai_target.py
# Conflicts: # pyrit/backend/README.md # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/targets.py # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py
# Conflicts: # frontend/src/components/Config/CreateTargetDialog.tsx # frontend/src/types/index.ts
# Conflicts: # frontend/src/components/Config/TargetTable.styles.ts # frontend/src/components/Config/TargetTable.tsx # frontend/src/types/index.ts # pyrit/backend/mappers/target_mappers.py # tests/unit/backend/test_mappers.py
… a saveability answer once its request goes away The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
…s, and auth mode only
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and local refusal scorers. A saved entry still can't hold a typed key.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description
This is one of the changes for saving Registry entries across restarts. It builds on #4051, which saves converters and scorers created through the API. Until the earlier PRs merge, the diff here also shows their changes.
The New Converter dialog only made converters that are gone after the backend restarts, wherever it opens: the Converters tab, the converter panel in chat, and New converter in the technique editor. It now saves them by default, and the Converters and Scorers tables say which entries come back.
base64_converter. Keeping the_converterending stays clear of the names the startup scripts register (base64,rot13), which would stop a saved converter of that name from restoring. The name field checks the backend's rule for saved names. A name the list already has, or one the backend says is taken (409), shows on the field, and focus moves there.role="status"so screen readers read it out, and target options mark Temporary targets.GET /api/converters/typesaddsnot_saveable_reasonto each type, from the checks a save runs, including the one for types that need an uploaded file. The dialog keeps no copy of those rules.Config/TargetOriginBadge.tsxandutils/targetOrigin.tstoRegistry/OriginBadge.tsxandutils/registryOrigin.ts. The target files change imports only.Screenshots: a new converter saves by default, a converter that uses a Temporary target is offered as Temporary with the reason, and the Converters table before and after a restart.




Not covered: