Skip to content

FEAT: Save targets created through the API (part 8) - #4026

Merged
varunj-msft merged 251 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Targets
Nov 17, 2026
Merged

varunj-msft merged 251 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Targets

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 8 of saving Registry entries across restarts. Parts 1 to 7 are merged, so the diff here is only this PR's changes. Part 9 (#4027) builds on it.

Design: 4.1 What gets saved, 1. The contract. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4025 (merged), which refuses saved targets that would authenticate with a key.

Problem. A target created through POST /api/targets is gone when the backend restarts. Restore (#3909) brings saved entries back at every start, but nothing saves one yet, and the target list doesn't say which targets will come back.

Change.

  • Saving is a registry library operation, so Python code saves a target the same way the API does. In pyrit/registry/registry_recipe_save.py (exported from pyrit.registry), prepare_saved_params_async turns off the values a target would take from this server and refuses what a saved entry can't hold, using the same checks restore does, including which targets it uses come back after a restart. It also accepts only what restore's build accepts: a setting the target would read from this server's environment, such as an OpenAI model name, has to be set, and a parameter the create API can't take, such as httpx_client_kwargs, is refused (400) before anything is written. save_then_register_async writes the recipe only if the name is free, registers the entry as saved, and deletes the recipe again if registering fails, but only the version it wrote, so a file another writer saved since is left alone. A save that's cancelled finishes, or is rolled back, before the cancellation goes on. The backend only passes the signed-in owner and the configured source, builds the response, and maps the library's errors to 400, 409 and 503.

    from pyrit.models.catalog import ComponentRecipe, RegistryRecipeKind
    from pyrit.registry import TargetRegistry, prepare_saved_params_async, save_then_register_async
    
    registry = TargetRegistry.get_registry_singleton()
    cls = registry.get_class("OpenAIChatTarget")
    params = await prepare_saved_params_async(
        registry=registry, type_name="OpenAIChatTarget", cls=cls, auth_mode="identity",
        params={"endpoint": "https://contoso.openai.azure.com/openai/v1", "model_name": "gpt-4o"},
    )
    target = await registry.create_instance_async(name="OpenAIChatTarget", params={**params, "auth_mode": "identity"})
    recipe = ComponentRecipe(class_module=cls.__module__, class_name="OpenAIChatTarget", auth_mode="identity", params=params)
    await save_then_register_async(
        kind=RegistryRecipeKind.TARGETS, name="contoso_gpt4o", recipe=recipe,
        register=lambda metadata: registry.instances.register(target, name="contoso_gpt4o", metadata=metadata),
    )
  • POST /api/targets takes "persist": true, which needs register left on and a name that starts with a lowercase letter and has at most 64 lowercase letters, digits and underscores (otherwise 422). The target is built as before, saved as a recipe under registry_recipes_source (by default ~/.pyrit/registry_recipes, so no Azure is needed) with the signed-in user recorded as the one who saved it, and then registered. If registering fails, the file is deleted again; if that delete fails too, the request gets 503 and the server log names the target and its folder, so the file can be removed.

  • Before saving, an OpenAI target's empty headers is set to "{}", so a saved one doesn't send this server's OPENAI_ADDITIONAL_REQUEST_HEADERS. Then the request goes through the same check restore uses, so a target that would hold or use a key, or leave an address to this server's settings, gets 400, and the message says it can still be created without saving it. A saved target can use saved targets and targets from configuration, which keep their own sign-in, but not a Temporary target, such as a round-robin member that won't come back after a restart.

  • Each target type declares the settings it reads from this server's environment when they're empty (environment_backed_parameters on PromptTarget). That replaces the backend's table keyed by class name, so GET /api/targets/types marks the same fields required that a save checks.

  • A name that a registered target or a saved file already has gets 409. The file is written create-only, so two saves of one name save one target. Storage that can't be written gets 503 and the target isn't registered. The server log names the target and its folder, with the error behind it and its chain (URL credentials removed, as restore logs), in case the write reached storage anyway. A failed rollback is logged the same way.

  • A target type that a scan's launch scripts added, or put in place of another class under an existing name, can't be saved (400): restore couldn't rebuild it after a restart, and with the backend staying not ready until every saved entry restores, that file would keep it down.

  • Each listed target has an origin: saved, configuration (registered by the server's startup scripts), or temporary (gone after a restart or reinitialization, such as a target created without persist or registered by a scan's launch scripts). Capability detection returns it too.

  • The seeded browser end-to-end runs, and the backend end-to-end runs when they start their own backend, point registry_recipes_source at a new temp folder, so they don't read or write the targets saved on the machine running them.

What stays the same. Creating a target without persist works as before, and permission checks are unchanged and come first, so a non-admin whose target would use a server credential still gets 403. A save runs inside the create request, and apply already reports busy while a request is in flight, so no new lock is needed. Restore is unchanged, except that it first marks the targets the startup scripts registered as configuration.

Not covered.

Tests and Documentation

  • tests/unit/registry/test_registry_recipe_save.py (no backend imports): a target built from prepared parameters is saved, with "headers": "{}", and registered as saved; a typed key, an empty key, a header or URL that can carry a credential, a class a launch script added, and a Temporary target it uses are refused, also when the class's metadata isn't cached; the recipe is deleted when registering fails, and only a taken name is a conflict; a setting left to the server's environment and a parameter the API can't take are refused; storage failures are reported as unavailable, and the log names the target and its folder with the cause, URL credentials removed; a failed undo is reported as storage trouble and logged with its cause; rollback leaves a file another writer replaced; a cancelled save finishes registering, or rolls back, before the cancellation goes on.
  • tests/unit/backend/test_target_service.py: an Entra ID target is saved and restore rebuilds it; a no-key target is saved without an owner when sign-in is off; a target that can't be saved is neither created nor saved; a taken name is a conflict only when saving; a saved file with the name is a conflict, stays as it was, and the built target is released; two saves of one name save one target; a round-robin is saved only over targets that come back after a restart; a class registered under another name is saved so restore finds it; a reserved name is refused as invalid, not as taken; the list shows where each target comes from; the administrator gate refuses an Entra ID Blob target before construction, with and without persist, and nothing is registered or written.
  • tests/unit/backend/test_api_routes.py: a save conflict maps to 409 and a storage failure to 503; the saved-entries source and the signed-in owner (or none when sign-in is off) reach the service; persist needs register and a valid name.
  • tests/unit/backend/test_mappers.py: the origin of each registered target, and on the instance.
  • tests/unit/backend/test_target_detection.py: detection returns a saved target's origin, and none for its inner targets.
  • tests/unit/backend/test_runtime_lifecycle.py: startup marks the targets its scripts register but not saved ones; a save in flight makes apply report busy.
  • tests/unit/registry/test_target_registry.py: a class a launch script adds, or puts in place of another under an existing name, is temporary.
  • tests/unit/registry/test_instance_registry.py and test_registry_recipe_storage.py: startup entries are marked and keep the mark when replaced; a recipe is created once, lists back with its owner and version, and is deleted once.
  • Removing each guard in turn fails at least one of these tests, including this round's: the restore-input and environment checks, the wait for a cancelled save, the versioned rollback, the logged cause and its redaction, and the origin in detection.
  • python -m pytest -n 8 tests/unit/converter tests/unit/prompt_target tests/unit/registry tests/unit/backend: 11,883 passed, 9 skipped, 4 xfailed. The cancellation behavior was also checked on Python 3.11 and 3.14, including asyncio.timeout turning the passed-on cancellation into TimeoutError.
  • Frontend npm run lint and npm run type-check (for playwright.config.ts): passed.
  • E2E_SEEDED_MODE=true npx playwright test --project=seeded --workers=1: 108 passed, with the backend started from the generated temp config.
  • ruff (with main's new RUF and F821 rules), ty, and the repo's keyword-only, async-suffix, public-import and docs checks: passed.
  • Run against a real backend process (this round, on part 9's head, which includes this one): an Entra ID target saves (201), httpx_client_kwargs gets a plain 400 rather than a can't-be-saved one, and one recipe file is written.
  • Earlier, run against a real backend with no environment files, so no server keys: an Entra ID target on an Azure OpenAI endpoint saves (201, origin saved); the same name gets 409; a typed key, an empty key and an endpoint with ?sig= get 400 without echoing the key; exactly one recipe file is written, with no key. After a restart the saved target is back and answers a chat message, and the Temporary one is gone.
  • pyrit/backend/README.md and doc/getting_started/pyrit_conf.md: how to save a target, the name rule, what it can hold, the headers default, the OpenAI SDK settings that still apply, and the status codes.

No JupyText changes.

The target catalog now applies the path rule to required parameters too, so a
target that needs a disallowed path is left out instead of listed without the
field, matching the scorer catalog. allowed_path_params is typed frozenset[str]
so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its
server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts:
#	tests/unit/backend/test_converter_service.py
A suite launch runs its launch scripts during preflight and sends its scans without them, so the clash is
raised there. Map it to 409 with its message, as a single launch does, and drop the per-entry message that
a clash can no longer reach.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep the useful persist/origin feature, but move its reusable persistence operation into the registry library before inclusion. The new _prepare_saved_params_async, _refuse_temporary_references, and _save_and_register_async helpers in pyrit/backend/services/target_service.py:738-876 own durable-parameter preparation, which registered dependencies may survive a restart, and save/register/rollback. The registry currently exposes the storage primitives, so a notebook caller wanting to save the same target must depend on backend service/request types or reconstruct those rules and compensation steps. The library-first framework guidance in #4175 assigns this behavior to the library and keeps backend services as adapters. Extract this operation using the existing registry, recipe models, rules and storage; leave authenticated-owner/configured-source extraction, HTTP error mapping, and runtime/process coordination in the backend. No generic persistence framework or redesign of the already-reviewed predecessor is needed.

This inclusion recommendation was independently verified at 48bc6d14c756d00990dc8ff4b6ab09bcaa0d1828. Implementation review is paused at that gate; this is not a full source review or approval.

# Conflicts:
#	pyrit/prompt_target/common/prompt_target.py
#	pyrit/prompt_target/hugging_face/hugging_face_chat_target.py
# Conflicts:
#	tests/unit/backend/test_runtime_lifecycle.py
@varunj-msft

Copy link
Copy Markdown
Contributor Author

Thanks, that makes sense. Saving now lives in the registry library (pyrit/registry/registry_recipe_save.py, exported from pyrit.registry): prepare_saved_params_async applies the same rules restore does, including which dependencies come back after a restart, and save_then_register_async writes the recipe, registers the entry, and undoes the write if registering fails. The backend just passes the signed-in owner and configured source, builds the response, and maps errors to HTTP. The description has a short notebook example, and the library tests save a target without the backend.

I made the same move up the stack, so converters, scorers, techniques and copies save through the library too, and the backend only keeps the lock that orders saves and deletes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this head with complete source coverage and independent finding verification. Five new issues remain below; the existing rollback concern also remains open. The main blocker is that the public Python save workflow accepts recipes that the same-head restore path cannot rebuild. No approval or source changes.

Comment thread pyrit/registry/registry_recipe_save.py Resolved
Comment thread pyrit/registry/registry_recipe_save.py Resolved
Comment thread pyrit/registry/registry_recipe_save.py Resolved
Saving now refuses what restore can't rebuild: a setting left to the server's environment (each target type
declares these, replacing the backend's class-name table) and parameters the create API can't set. A cancelled
save finishes or is undone before the cancellation goes on, rollback deletes only the version this save wrote,
storage failures log a redacted cause chain, and capability detection keeps the target's origin.
Saved-entry storage failures now pass their traceback through redact_urls, and code scanning flags the scheme pattern
as slow on a long run of letters. A scheme is short, so bounding it keeps the match linear.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@varunj-msft
varunj-msft merged commit 745447e into microsoft:main Nov 17, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants