Skip to content

FEAT: Restore saved registry entries at startup and stay not ready if any fail (part 4) - #3909

Merged
varunj-msft merged 136 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Restore-Saved-Entries
Nov 12, 2026
Merged

varunj-msft merged 136 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Restore-Saved-Entries

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 4 of saving Registry entries across restarts. Stacked on part 3 (#3908); parts 5 (#3965) and 6 (#3986) build on it. This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Rebuild-Saved-Entries...varunj-msft/13178-Restore-Saved-Entries

Design: 4.6 When a saved entry can't be restored, 4.2 Where recipes live. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #3908. Until the earlier PRs merge, the diff here also shows their changes.

Problem. Saved entries need to come back at startup and after a live apply. A broken one must stop runs instead of vanishing; otherwise a scan that expects a saved converter or adversarial_chat would run with something else, or fail later.

Change.

  • New setting registry_recipes_source: a local folder or Azure Blob location, default ~/.pyrit/registry_recipes. It's validated like scenario_presets_source and documented in pyrit_conf.md and .pyrit_conf_example.
  • RuntimeLifecycle restores saved entries at startup, after the initializers run, and on apply, after the registries are reset. The files are read in a worker thread, and the entries are built on the runtime loop, because a constructor such as HuggingFaceChatTarget's starts a task on it.
  • Each entry is built through the API's own create for its kind (pyrit/backend/services/saved_entry_restore.py calls the target, converter, scorer and technique services), so a saved entry passes the same checks as a new one. Restore runs as the server and can't tell who wrote a file, so targets are built as an administrator would build them, with this server's upload folder.
  • If any entry can't be restored, the runtime stays failed with outcome restore-failed and doesn't become ready. The readiness gate already answers 503 on every route except configuration, initializers, auth, health and runtime status, and /api/health shows ready: false, so only the configuration pages work until it's fixed. A startup script that fails already works this way.
  • Apply now also runs from failed. It resets the registries and restores again, so an administrator can fix or delete the listed files and apply instead of restarting. The failure message says to apply or restart when live apply is on, and to restart when it's off; applying with live apply off says to restart.
  • Each failure is logged once with its message and traceback, with URL credentials removed, and a summary line says how many entries couldn't be restored. For a file that fails validation, that log record also lists every field and what's wrong with it, in full, escaped and without the stored values; the runtime status keeps the short summary, which now drops URL credentials before it's shortened. GET /api/config/runtime lists restore_failures and restore_ignored.
  • The docs say that write access to this source is administrator authority on the backend, and to use a dedicated folder or blob prefix.
  • The backend unit tests, the analytics benchmark fixture and the offline browser fixture point the recipes folder at a temp folder, so they don't read the developer's own saved entries.

What stays the same. With no saved entries, startup and apply behave as before. A file whose name isn't a valid entry name is listed as ignored and doesn't block readiness. Apply still refuses restart-required.

Next. #3965 (part 5) lists the entries that couldn't be restored on the Configuration page, for administrators, and shows everyone else one line about it. #4071 (part 20) adds Delete to that list.

Tests and Documentation

  • tests/unit/backend/test_saved_entry_restore.py (new): each kind restores through its create; a target its create refuses fails only that entry; a target that uploads files gets this server's upload folder, and fails where the server sets none; a technique its create request refuses is reported as invalid.
  • tests/unit/backend/test_runtime_lifecycle.py: startup restores an entry that uses a target the startup scripts register; the files are read off the runtime loop and the entries are built on it; an apply after a startup restore brings back the same converter and scorer, with the real registry reset; an entry that can't be restored keeps the app not ready, with 503 outside configuration and ready: false in /api/health; apply from failed restores again once the file is fixed, and with live apply off it says to restart; each failure is logged once without URL credentials; an unreadable folder gets its own message; ignored files don't count as failures; fixing a file and restarting restores it; cancellation still propagates.
  • tests/unit/registry/test_registry_recipe_restore.py: a file that fails validation logs every problem in full in its one error record, including many unknown fields and long keys that share a prefix, escaped, without the stored values or URL credentials, and the short summary has no URL credentials either.
  • frontend/src/components/Configuration/Reinitialize.test.tsx: Reinitialize stays available while a restore failed. tests/unit/setup/test_configuration_loader.py covers the new setting.
  • pytest tests/unit/backend tests/unit/registry tests/unit/setup tests/unit/common -n 4: 6,383 passed, 9 skipped, 4 xfailed. jest src/hooks src/App.test.tsx src/components/Layout src/components/Registry src/components/Configuration: 820 passed. tsc --noEmit passed. After the validation-log change: pytest tests/unit/registry tests/unit/backend -n 8: 5,326 passed, 9 skipped, 4 xfailed.
  • Checked locally with ruff, ruff format and ty on the changed files (the pre-commit hooks can't install in my environment, so CI runs them).
  • Docs: doc/getting_started/pyrit_conf.md and .pyrit_conf_example describe registry_recipes_source, who should be able to write to it, and that the backend doesn't become ready until every saved entry restores: fix or delete the listed files, then apply or restart. JupyText: not run (no notebooks changed).

…API input

External construction now rejects a parameter whose type includes Path unless the
calling service passes the name in allowed_path_params after making the value safe.
The converter service allows its upload parameters once they are stored, and the
target service allows the server-set upload folder and the GitHubCopilotTarget
working directory. The scorer-only check is removed, and the "Valid parameters"
list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a
target that needs a disallowed path is left out instead of listed without the
field, matching the scorer catalog. allowed_path_params is typed frozenset[str]
so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its
server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts:
#	tests/unit/backend/test_converter_service.py
@varunj-msft varunj-msft changed the title FEAT: Restore saved registry entries at startup and apply (part 4) FEAT: Restore saved registry entries at startup and stay not ready if any fail (part 4) Nov 10, 2026
@varunj-msft

Copy link
Copy Markdown
Contributor Author

Thanks Rich, no worries! It's back to not ready, with your follow-up included:

Since one bad file now keeps the app down, saving also refuses a couple more things restore would (#4026, #4066), and editing a broken entry just saves its file (#4072). On your #3908 note about converters: #4051 already refuses saving a converter that sets any file parameter, so saving and restore can't disagree there.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Please address the P3 on schema-validation diagnostics before merging; that change doesn't need another review.

The runtime status keeps its short summary. The restore log now also lists each field's full location
and message, escaped, without the stored values, and with URL credentials removed. Credentials are
removed before a location is shortened, so the summary can't keep part of one either.
Its text can carry a SAS URL. The 503 handler no longer logs it as a debug traceback, and the scenario
suite service no longer attaches it to its warning; both log the storage message, which has no credentials.
@varunj-msft
varunj-msft merged commit 1f5f583 into microsoft:main Nov 12, 2026
64 of 66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants