Repository navigation
FEAT: Save converters and scorers created through the API (part 12) - #4051
Open
varunj-msft wants to merge 438 commits into
Open
varunj-msft wants to merge 438 commits into
varunj-msft wants to merge 438 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Saved LocalViolenceClassifierScorer instances can still download models using the server’s cached Hugging Face token or HF_TOKEN, with no save-policy refusal or safeguard disabling authenticated downloads.
4 open issues
What changed in this PR
Extends PyRIT’s saved-target infrastructure to converters and scorers created through the API, allowing eligible registry entries to survive backend restarts.
Changes:
- Adds optional persistence with shared validation, dependency checks, ownership metadata, and rollback on registration failure.
- Restores saved entries during startup and reinitialization, reports their origin, and protects them from replacement.
- Includes target-origin and ownership UI, isolated test storage, regression coverage, and configuration documentation.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging safeguards. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement refusal. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe-storage configuration. |
| tests/unit/registry/test_instance_registry.py | Tests origin metadata and replacement guards. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique collision handling. |
| tests/unit/backend/test_scorer_service.py | Covers scorer persistence and restoration. |
| tests/unit/backend/test_scorer_routes.py | Covers persistence requests and HTTP errors. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch conflict reporting. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_saved_entries.py | Tests persistent-reference eligibility. |
| tests/unit/backend/test_mappers.py | Tests target origin and ownership mapping. |
| tests/unit/backend/test_initializer_service.py | Tests unavailable local storage responses. |
| tests/unit/backend/test_auth_middleware.py | Tests signed-in user ID extraction. |
| tests/unit/backend/test_analytics_benchmark.py | Verifies benchmark storage isolation. |
| tests/unit/backend/conftest.py | Isolates saved recipes in backend tests. |
| tests/end_to_end/conftest.py | Isolates launched backend recipe storage. |
| test_support/analytics_benchmark.py | Configures benchmark-specific recipe storage. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry collisions during tool setup. |
| pyrit/setup/initializers/targets.py | Guards target grouping against saved collisions. |
| pyrit/setup/initializers/scorers.py | Guards automatic tagging of saved scorers. |
| pyrit/setup/configuration_loader.py | Adds recipe-storage source configuration. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares the scorer unsuitable for persistence. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares credential-dependent persistence refusal. |
| pyrit/score/scorer.py | Adds scorer persistence-policy metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares server credential fallback. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares credential-dependent persistence refusal. |
| pyrit/registry/registry_recipe_storage.py | Implements per-registry JSON recipe storage. |
| pyrit/registry/instance_registry.py | Tracks origins, owners, and replacement refusals. |
| pyrit/registry/components/scorer_registry.py | Reserves the types instance name. |
| pyrit/registry/components/attack_technique_registry.py | Adds saved metadata and collision protection. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares persistence refusal for server sign-in. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/openai/openai_target.py | Declares credential, header, and destination policies. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token fallback and persistence refusal. |
| pyrit/prompt_target/http_target/http_target.py | Refuses persistence of raw HTTP requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallbacks. |
| pyrit/prompt_target/github_copilot_target.py | Declares GitHub token fallback. |
| pyrit/prompt_target/common/prompt_target.py | Defines target persistence-policy metadata. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and endpoint policies. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares token and container-URL policies. |
| pyrit/prompt_target/a2a_target.py | Declares destination parameters for validation. |
| pyrit/models/catalog/target.py | Adds target origin and ownership fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin. |
| pyrit/models/catalog/registry_recipe.py | Defines recipes, origins, owners, and restore reports. |
| pyrit/models/catalog/__init__.py | Exports recipe-related models lazily. |
| pyrit/converter/converter.py | Adds converter credential-fallback metadata. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Declares Azure Speech credential fallback. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Declares Azure Speech credential fallback. |
| pyrit/backend/services/scorer_service.py | Saves scorers and exposes their origins. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes actionable saved-entry launch conflicts. |
| pyrit/backend/services/scenario_run_service.py | Detects launch scripts conflicting with saved entries. |
| pyrit/backend/services/saved_entries.py | Shares validation, saving, registration, and rollback. |
| pyrit/backend/services/runtime_lifecycle.py | Restores recipes and reports startup failures. |
| pyrit/backend/services/converter_service.py | Saves converters and guards saved-entry deletion. |
| pyrit/backend/routes/targets.py | Passes persistence context and maps save errors. |
| pyrit/backend/routes/scorers.py | Adds scorer persistence response handling. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage diagnostics and list failures. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage diagnostics and list failures. |
| pyrit/backend/routes/initializers.py | Handles local storage failures safely. |
| pyrit/backend/routes/converters.py | Maps persistence and deletion errors. |
| pyrit/backend/routes/common.py | Shares owner extraction and save-refusal responses. |
| pyrit/backend/README.md | Documents persistence APIs and restrictions. |
| pyrit/backend/models/targets.py | Adds persistence validation and saveability fields. |
| pyrit/backend/models/scorers.py | Adds validated scorer persistence requests. |
| pyrit/backend/models/converters.py | Adds persistence validation and origin fields. |
| pyrit/backend/middleware/auth.py | Exposes the current user’s object ID. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and saver information. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin. |
| frontend/src/utils/targetOrigin.ts | Defines target-origin display labels. |
| frontend/src/utils/registryName.ts | Generates valid, available registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds target persistence and ownership types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and ownership fixtures. |
| frontend/src/services/errors.ts | Recognizes typed save refusals. |
| frontend/src/services/errors.test.ts | Tests save-refusal recognition. |
| frontend/src/components/Config/TargetTable.tsx | Displays origins, savers, and ownership filtering. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests origin display and ownership filtering. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles saver captions and filter controls. |
| frontend/src/components/Config/TargetOriginBadge.tsx | Renders target-origin badges. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/registry.spec.ts | Tests target saving through the GUI. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline fixture recipe storage. |
| frontend/e2e/config.spec.ts | Tests saving and ownership UI behavior. |
| frontend/e2e/_targets.ts | Extends target mocks with persistence metadata. |
| doc/gui/0_gui.md | Explains saved targets and ownership controls. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restoration behavior. |
| .pyrit_conf_example | Adds commented recipe-storage configuration. |
Review effort: Balanced
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 02:55
… text Recipes now share one format version, RECIPE_SCHEMA_VERSION, and a file without one still reads as version 1. A saved target keeps the auth mode its create request chose as its own field, so restore can replay that request; only a target may have one. Every failure message is escaped and shortened where it's built, so a stored key with a new line or a terminal escape, or a long version number, can't change a log or run on. A params value nested too deeply for pydantic is reported as nested too deeply, and the technique recipe says why it copies the create request's fields.
Restore still reads, checks, and orders the saved entries, but no longer builds them itself: restore_async takes the builder, so the backend can build each entry through the API's own create. The copies of the technique build, the reserved technique names, and the upload-folder refusal are gone. A name another entry holds is a name clash; a reserved name, or a technique name a scenario can't use, is an invalid recipe. Each failure message is escaped and shortened where it's built, and the tests use small test classes and a builder that raises instead of OpenAI's error text.
Startup and apply now restore saved entries through the API's create for each kind: a new backend builder turns each recipe back into its create request and calls the target, converter, scorer, or technique service. A target is built as an administrator would build it, with this server's upload folder.
…restore end to end The configuration docs now say that the backend rebuilds every saved entry as an administrator, so write access to the source is administrator authority, and say where ignored files are listed. The startup log says when a folder couldn't be read instead of asking to fix or delete files. New tests restore each kind through its create, check the Copilot token and upload folder rules, run an apply after a startup restore with the real registry reset, and check one log record per failure without a SAS token.
… a saveability answer once its request goes away The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
…s, and auth mode only
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and local refusal scorers. A saved entry still can't hold a typed key.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description
This is one of the changes for saving Registry entries across restarts. It builds on #4027, which is merged.
A converter or scorer created through the API is gone when the backend restarts. With
"persist": true,POST /api/convertersandPOST /api/scorersnow save it the wayPOST /api/targetsdoes (#4026), so restore brings it back at every start.target_service.pyto a sharedservices/saved_entries.py: turn off the server values an entry would read, refuse what it can't hold or a Temporary entry it uses, write the recipe create-only, register it, and delete the recipe again if registering fails. The owner and not-saveable helpers move toroutes/common.py, andTargetOrigin(new in FEAT: Save targets created through the API (part 8) #4026) becomesRegistryEntryOrigin. Targets behave as before.persiston converters and scorers follows the target rules. It needsregisterand a name matching^[a-z][a-z0-9_]{0,63}$(422 otherwise). A refusal is a typed 400 (/errors/not-saveable, with the cause innot_saveable_reason), a taken name is 409, and unreadable storage is 503. The owner is recorded in the saved file as for targets. As with targets, a private converter or scorer can't be saved (422), and a taken converter name is a 409 whether or not it's saved.@apply_defaults) is written into the recipe by name, each entry of a list included, so restore uses the same entries even if nothing sets that default at the next start. A default that is Temporary, or isn't a registered entry, is refused. Other configured defaults apply again from the configuration at every start, as they do for startup entries.sourcecan't be saved yet (422).origin(saved,configurationortemporary) on list, get and create, and the startup entries of all three kinds are marked as configuration.DELETE /api/converters/{id}returns 409 for a saved converter, since it would come back from its file at the next start. Temporary and configuration converters are removed as before.typesis now reserved, as it is for targets and converters, sinceGET /api/scorers/typeswould hide a scorer with that name.pyrit_conf.mddescribe saving converters and scorers.Not covered:
GET /api/scorers/typesdoesn't say yet why a scorer type can't be saved; saving one refuses it with the reason.