Skip to content

FEAT: Save targets from the Registry (part 9) - #4027

Merged
varunj-msft merged 302 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-From-Registry
Nov 19, 2026
Merged

varunj-msft merged 302 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-From-Registry

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 9 of saving Registry entries across restarts. Parts 1 to 8 are merged, so the diff here is only this PR's changes. Part 11 (#4050) builds on it, and so do the side branches part 16 (#4068) and part 19 (#4070).

Design: 1. The contract, 4.8 Names. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4026 (merged), which saves targets created through the API.

Problem. The New Target dialog in Registry > Targets still makes targets that are gone after the backend restarts. It sends no name, so the backend makes one up (compat_<hex>), and the Targets table doesn't say which targets will come back.

Change.

  • The dialog has a Registry Name field. Until the user edits it, it suggests a name from the type and model that no listed target has (openai_chat_gpt_4o). The model tells saved targets apart and keeps the suggestion off names the startup scripts register, such as openai_chat. The field checks the same rule the backend uses for saved names. A name the Targets list already has, or one the backend says is taken (409), shows on the field and holds Save until the name changes.
  • The dialog saves by default (persist: true). When a target can't be saved, the button reads Create as Temporary and a warning above the buttons gives the cause once, and that a Temporary target is gone after the backend restarts or is reinitialized. It sits outside the scrolling form, so choosing a sign-in never scrolls the choice out of view. The dialog says so before submitting. The backend's type list tells it when the type can't be saved or the chosen sign-in can't be saved (an API key, for now). For the values entered, such as a typed credential or a round robin with a Temporary member, it asks the new POST /api/targets/saveability, which runs create's parameter and permission checks and the library's save checks on the type, parameters, and auth mode, without building or saving anything, so the warning and the save share one policy. The check waits until typing pauses, and only its answer for the current values counts. An answer is dropped once the values change or the dialog closes, so the same values later get a fresh check. If Save is pressed first, the backend's refusal turns the button into Create as Temporary for those exact values, and changing any of them goes back to Save Target. No other error offers Temporary.
  • It keeps FEAT: Private targets that only their creator can see and use #4024's rules: saved targets are shared and keep their credentials, so a target set to Only me (chosen, or because it uses a private target) or with Allow updates on is created as Temporary. The dialog decides that itself and doesn't ask the backend; create still refuses persist with private or allow_credential_updates (422). Each warning gives only its cause ("Private targets stay Temporary.", "Saved targets can't use an API key yet."), since more than one can apply, so it never suggests that one change makes the target saveable.
  • A new target starts with Microsoft Entra ID, or No key, when a saved target of its type can use it, so the usual case saves. This also applies when the type's details arrive after the type was picked. Picking another type resets the choice and clears a typed key. For an OpenAI endpoint that isn't on Azure, the Entra ID error now says to choose No key for a server that needs no key, such as a local one. It's the Endpoint URL field's own error, shown once the field is left, not while the address is typed, so it stays next to the address on a narrow screen.
  • GET /api/targets/types adds saveable_auth_modes and not_saveable_reason to each type. They come from the registry library's get_class_saveability, the same class check a save runs (a class a scan's launch script added, the class check, and the server-setting check for each auth mode with those settings turned off), so neither the backend nor the dialog keeps a copy of the rule.
  • A refused save gets a typed 400 ("type": "/errors/not-saveable"), so the dialog doesn't read messages. Its detail keeps the full sentence for API clients, and not_saveable_reason gives the cause alone, as the type list and the saveability check do. A taken name is an untyped 409 whether or not the target is saved, so a Temporary target's taken name also shows on the name field. When the only problem is an endpoint that isn't a full http(s) or ws(s) address, the request gets a plain 400 instead, since a Temporary target with that address wouldn't work either.
  • The warning uses role="status" so screen readers read it out, and the Entra ID error is the field's validation message. Fluent's message bars only announce through an announcer the app doesn't render.
  • The Targets table labels each target Saved, From configuration or Temporary, and the round robin member list marks Temporary members.
  • The GUI doc says where saved targets are kept, how to remove one for now, and what happens if a startup script later registers the same name.

What stays the same. API callers that don't send persist get Temporary targets as before, and may still leave out the name. Permission checks still come first, and restore is unchanged.

Not covered.

  • Picking a server credential, and keeping typed keys in a key vault. Until then a target that uses an API key is Temporary.
  • Saving converters, scorers and techniques, and deleting saved entries. Those come in parts 12 to 14 and part 20.

Tests and Documentation

  • tests/unit/registry/test_registry_recipe_rules.py: check_saveable_class lists the auth modes a saved target of each type can use, refuses a class that can never be saved, and agrees with the check a save runs for every registered target type and auth mode.
  • tests/unit/backend/test_target_service.py: the type list says which auth modes a saved target can use and why a type can't be saved; saving a type that can't be saved is refused; an endpoint that isn't a full web address is an error to fix, not a refusal.
  • tests/unit/backend/test_api_routes.py: a refused save is a 400 with "type": "/errors/not-saveable"; other bad requests stay untyped; POST /api/targets/saveability answers with the save checks' reason, and a request creating would refuse gets the same 400 or 403.
  • tests/unit/backend/test_target_service.py (TestCheckSaveability): the check answers as saving would for a Temporary round-robin member, a typed key and Entra ID, without building anything, and refuses a request creating would refuse.
  • CreateTargetDialog.test.tsx (new "CreateTargetDialog saving" block): saving by default with Entra ID or No key, also when the type details arrive late; the suggested name from the type and model until it's edited; names already listed or taken (409) held on the field; when Create as Temporary is offered and why (an API key or a type that can't be saved from the type list; a typed credential or a round robin with a Temporary member from the backend's check, and not from the list's labels), and only after a typed refusal for those exact values; an answer for values that have since changed is ignored; no Temporary offer on 409, 422, 403 or other errors; the warning as a live region; starting over after the dialog closes; the Entra ID error as the Endpoint URL field's description once the field is left; each reason as its cause alone (an API key with Only me, then Entra ID); the same values checked again after the dialog is reopened, even when that check fails; and a target only you can use never checked, on a controlled debounce clock.
  • TargetTable.test.tsx: the Saved, From configuration and Temporary labels. registryName.test.ts and errors.test.ts: the name suggestion and the typed refusal check.
  • frontend/e2e/config.spec.ts (mock): saving through the dialog sends Entra ID and persist: true. frontend/e2e/registry.spec.ts (seeded, real backend): a target saved from the dialog is still Saved after a reload.
  • Each of the 64 ways I tried to break the dialog, the labels or the saveability check fails at least one of these tests (from the earlier round).
  • This round (main with FEAT: Private targets that only their creator can see and use #4024 merged in, and the review changes): get_class_saveability gives the class answer and the save refusal the same cause; a taken name is an untyped 409 for both persist values; the saveability check and the not-saveable 400 give the cause alone; Only me and Allow updates create Temporary targets without asking the backend, and turning Allow updates off offers Save again; a failed saveability check keeps Save; the warning sits outside the scrolling form and nothing scrolls; a refusal at submit shows the cause once; the name hint shows the 64-character limit and hides while an error shows; a Temporary HTTPTarget's taken name shows on the name field.
  • Follow-up round (the four comments after that): each Temporary reason gives only its cause; a saveability answer is dropped once its values change or the dialog closes, and a closed dialog sends no check; the private-target test runs on fake timers. Without the hook's reset the reopen test fails, and if Only me sent a check the private-target test would fail every time. With the fix it passed 6 of 6 concurrent runs.
  • python -m pytest -n 6 tests/unit/registry tests/unit/backend/test_target_service.py tests/unit/backend/test_api_routes.py tests/unit/backend/test_private_targets.py tests/unit/backend/test_private_target_sends.py tests/unit/backend/test_saved_entry_restore.py tests/unit/backend/test_runtime_lifecycle.py: 2,222 passed, 5 skipped, 2 xfailed
  • cd frontend && npx jest CreateTargetDialog.test.tsx errors.test.ts api.test.ts: 278 passed
  • cd frontend && npx playwright test --project=mock e2e/config.spec.ts: 18 passed
  • npx tsc --noEmit, npx eslint on the changed files, and pre-commit (ruff, ty, docs checks): passed
  • Run against a real backend process (this round): POST /api/targets/saveability says an Entra ID target and a round robin over a saved target can be saved, gives the credential reason for a typed key without echoing it, and refuses a missing model with the same 400 as create.
  • Earlier, run against a real backend in a real browser, with the server using its own Entra ID credential: saved an Entra ID chat target on Azure OpenAI and a local No key target from the dialog. A target with an API key, a round robin using a Temporary target, and a target created through the API without persist came out Temporary, with the reason shown. After a backend restart the saved targets came back as Saved, the Temporary ones were gone, and a message sent through the restored Entra ID target got an answer. No key was in any saved file.
  • doc/gui/0_gui.md (Registry): saving a target, where saved targets are kept, how to remove one for now, and what happens when a startup script registers the same name. pyrit/backend/README.md: the new type fields, the typed 400, and the saveability check, including the create checks it doesn't run.

No JupyText changes.

The banner tied a retry to its failure only through the failed state, so a
retry from an earlier failure could still show its answer when the polls
missed the runtime becoming ready in between. The generation changes only
when the runtime becomes ready or the backend restarts, so a new one now
starts a new failure too.
A long failure list made the banner taller than a phone screen, and the app
doesn't scroll, so Retry could end up out of reach. The list now scrolls on
its own within a third of the screen, and can be scrolled from the keyboard.
When the list can't be loaded, the banner says so, keeps the last list it had,
and keeps trying.
Scripts a scan runs at launch register their targets, converters and scorers
with replace, so one that used a saved entry's name silently replaced it for
everyone, and scripts that skip names already taken (the technique and tools
scripts, round-robin grouping, the scorer script's preferred scorers) left the
scan with the saved entry instead of their own. Restore now marks the entries
it rebuilds, the registries and those scripts refuse to replace or stand in for
a marked entry, and the launch stops with a 409 naming the script and the entry,
even when the script caught the refusal itself. A suite shows the same message
for that entry.
A launch script that caught the refusal and raised its own error skipped the
recorded-refusal check, so the launch failed with that error instead of the
409 naming the saved entry. The clash is now reported on any error once a
refusal was recorded; other errors pass through as before.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found six issues to address before approval. Details are on the affected lines.

Comment thread pyrit/registry/registry_recipe_save.py Resolved
Comment thread pyrit/registry/registry_recipe_save.py Resolved
Comment thread pyrit/registry/registry_recipe_save.py Resolved
Saving now refuses what restore can't rebuild: a setting left to the server's environment (each target type
declares these, replacing the backend's class-name table) and parameters the create API can't set. A cancelled
save finishes or is undone before the cancellation goes on, rollback deletes only the version this save wrote,
storage failures log a redacted cause chain, and capability detection keeps the target's origin.
The dialog keeps the type's own reasons from the type list and gets the rest from POST /api/targets/saveability,
which runs create's checks and the library's save checks without building or saving anything, so the warning and
the save share one policy. The Entra ID endpoint message shows on the Endpoint URL field after it loses focus.
Saved-entry storage failures now pass their traceback through redact_urls, and code scanning flags the scheme pattern
as slow on a long run of letters. A scheme is short, so bounding it keeps the match linear.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of this revision: three P2 concerns need changes (library ownership of save policy, Temporary name-conflict error mapping, and recovery instructions for dependent round robins). Six P3 improvements are nonblocking. Findings were independently checked against this head.

Comment thread doc/gui/0_gui.md Resolved
Comment thread pyrit/backend/README.md Resolved
Comment thread pyrit/backend/models/targets.py Resolved
Comment thread frontend/src/hooks/useSaveability.ts Resolved
… a saveability answer once its request goes away

The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@varunj-msft
varunj-msft merged commit 4e8aa59 into microsoft:main Nov 19, 2026
65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants