Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
389 commits
Select commit Hold shift + click to select a range
6fb1e00
Merge origin/main
varunj-msft Nov 8, 2026
405c2bd
Import from public packages, as main now requires
varunj-msft Nov 8, 2026
d35eea9
Merge origin/main
varunj-msft Nov 8, 2026
9afbd2c
Merge origin/main
varunj-msft Nov 8, 2026
23d7976
Merge origin/main
varunj-msft Nov 8, 2026
91c76d0
Import from public packages, as main now requires
varunj-msft Nov 8, 2026
4ded6e6
Merge origin/main
varunj-msft Nov 8, 2026
13bd3ab
Check a target that takes no credentials as the default auth mode
varunj-msft Nov 8, 2026
2b97d83
Merge origin/main
varunj-msft Nov 8, 2026
aaf2c15
Merge origin/main
varunj-msft Nov 8, 2026
cbb71f9
Merge origin/main
varunj-msft Nov 8, 2026
d0499fa
Filter the table test by the fixture's target type, which main change…
varunj-msft Nov 8, 2026
da9ce20
Merge origin/main
varunj-msft Nov 8, 2026
622bbb8
Report storage failures with one error and answer them with 503
varunj-msft Nov 9, 2026
1cae0b9
Merge the single storage error
varunj-msft Nov 9, 2026
eba7bf2
Escape file names once and simplify the saved recipe models
varunj-msft Nov 9, 2026
9f24124
Merge the recipe model simplification
varunj-msft Nov 9, 2026
2bbfc8f
Report restore bugs as build errors and log each failure once
varunj-msft Nov 9, 2026
3735dbc
Merge the restore logging and build error fixes
varunj-msft Nov 9, 2026
ca7563d
Stay ready when a saved entry can't be restored, and hold its name
varunj-msft Nov 9, 2026
a9dfc86
Merge the stay-ready restore
varunj-msft Nov 10, 2026
67416f1
Reset the registries between runtime tests, as a real startup does
varunj-msft Nov 10, 2026
e6e419e
Merge the runtime test isolation
varunj-msft Nov 10, 2026
81c0317
Stop a launch whose script takes the name of a saved entry that failed
varunj-msft Nov 10, 2026
6ee84f0
Merge origin/main
varunj-msft Nov 10, 2026
d43a7e8
Merge origin/main
varunj-msft Nov 10, 2026
0664063
Merge origin/main
varunj-msft Nov 10, 2026
3b3ac59
Merge origin/main
varunj-msft Nov 10, 2026
04fe60f
Merge origin/main
varunj-msft Nov 10, 2026
6911327
Merge origin/main
varunj-msft Nov 10, 2026
ee45962
Refuse OpenAI targets while the server sets OPENAI_CUSTOM_HEADERS
varunj-msft Nov 10, 2026
6a98d17
Merge origin/main
varunj-msft Nov 10, 2026
4a3f2e5
Use final origin names, record startup entries at registration, and t…
varunj-msft Nov 10, 2026
9f5d390
Merge the saved target origin and naming changes
varunj-msft Nov 10, 2026
bf480f5
Merge the saved target origin and naming changes
varunj-msft Nov 10, 2026
05f9bd1
Merge the saved target origin and naming changes
varunj-msft Nov 10, 2026
51d4fff
Explain that a credential refusal also covers using this server's cre…
varunj-msft Nov 10, 2026
a3d4f92
Merge the credential explanation
varunj-msft Nov 10, 2026
3c7a918
Merge the credential explanation
varunj-msft Nov 10, 2026
452ba54
Merge the credential explanation
varunj-msft Nov 10, 2026
8ff7bda
Merge the credential explanation
varunj-msft Nov 10, 2026
8614025
Use one recipe format version, keep the auth mode, and escape failure…
varunj-msft Nov 10, 2026
5a3f18b
Merge the recipe format changes
varunj-msft Nov 10, 2026
b3c7847
Build restored entries with a builder the caller supplies
varunj-msft Nov 10, 2026
af4075f
Merge the restore builder changes
varunj-msft Nov 10, 2026
061807a
Warn that writing saved entries is administrator authority, and test …
varunj-msft Nov 10, 2026
1740964
Merge the restore builder changes
varunj-msft Nov 10, 2026
01d99d7
Merge the restore builder changes
varunj-msft Nov 10, 2026
600717f
Merge the restore builder changes
varunj-msft Nov 10, 2026
095e76b
Merge the restore builder changes
varunj-msft Nov 10, 2026
0bf773d
Merge the restore builder changes
varunj-msft Nov 10, 2026
7a547aa
Say restore can't tell who wrote a saved file
varunj-msft Nov 10, 2026
f71259b
Merge the saved file wording
varunj-msft Nov 10, 2026
8902768
Merge the saved file wording
varunj-msft Nov 10, 2026
339fd15
Merge the restore builder changes
varunj-msft Nov 10, 2026
f1dcab1
Merge the saved file wording
varunj-msft Nov 10, 2026
986cc90
Test the parser's depth limit without depending on the stack size
varunj-msft Nov 10, 2026
1096925
Merge the parser depth test fix
varunj-msft Nov 10, 2026
f25b520
Merge the parser depth test fix
varunj-msft Nov 10, 2026
1845b64
Merge the parser depth test fix
varunj-msft Nov 10, 2026
bd2224b
Merge the parser depth test fix
varunj-msft Nov 10, 2026
b454e75
Merge the parser depth test fix
varunj-msft Nov 10, 2026
83a0bca
Merge the parser depth test fix
varunj-msft Nov 10, 2026
bd03690
Merge the parser depth test fix
varunj-msft Nov 10, 2026
2e9cbb0
Merge the parser depth test fix
varunj-msft Nov 10, 2026
fb90af8
Read saved entries in a worker thread when restore isn't given what w…
varunj-msft Nov 10, 2026
c44d2ce
Merge the worker-thread read
varunj-msft Nov 10, 2026
abd116a
Stay not ready when a saved entry can't be restored, and let apply retry
varunj-msft Nov 10, 2026
0bfa6cc
Drop the refusal of held names from launches
varunj-msft Nov 10, 2026
c62118f
Merge the not-ready restore
varunj-msft Nov 10, 2026
c0aa46d
Merge the not-ready restore
varunj-msft Nov 10, 2026
f59306e
Read saved recipes off the event loop by default, and harden URL reda…
richlundeen Nov 10, 2026
7d90d68
Merge the not-ready restore
varunj-msft Nov 10, 2026
0f0c53d
Treat a class a launch script puts in place of another as temporary
varunj-msft Nov 10, 2026
26e1d0e
Merge the not-ready restore
varunj-msft Nov 10, 2026
2ea542b
Merge the not-ready restore
varunj-msft Nov 10, 2026
9fca651
Merge the not-ready restore
varunj-msft Nov 10, 2026
f1ff143
Keep an "@" in a URL's path when redacting URLs in text
richlundeen Nov 10, 2026
009e761
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
21f22a4
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
e9103ff
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
57b069a
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
7c88228
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
616a7df
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
fede632
Merge the restore read and redaction fixes
varunj-msft Nov 10, 2026
cc4ae4d
Merge main
varunj-msft Nov 10, 2026
9f4c71e
Merge main
varunj-msft Nov 10, 2026
5121026
Merge main
varunj-msft Nov 10, 2026
d15f7fd
Merge main
varunj-msft Nov 11, 2026
84ad3fe
Merge main
varunj-msft Nov 11, 2026
ddfe03c
Merge main
varunj-msft Nov 11, 2026
7104a25
Merge main
varunj-msft Nov 11, 2026
be0b885
Use the dialog's save button in the new tests from main
varunj-msft Nov 11, 2026
7b08ca0
Press the hidden-targets button in the saved-by test
varunj-msft Nov 11, 2026
6c8d33a
Merge the dialog test fixes
varunj-msft Nov 11, 2026
cabb56f
Merge the dialog test fixes
varunj-msft Nov 11, 2026
187191c
Answer suite readiness storage failures through the shared storage ha…
varunj-msft Nov 11, 2026
ee8162c
Give the HTTPTarget save test the connection field a create now needs
varunj-msft Nov 11, 2026
bb2a620
Merge the readiness and save test fixes
varunj-msft Nov 11, 2026
db7451a
Merge the readiness and save test fixes
varunj-msft Nov 11, 2026
b200c92
Merge the readiness and save test fixes
varunj-msft Nov 11, 2026
4c42a6b
Merge the readiness and save test fixes
varunj-msft Nov 11, 2026
43096e8
Merge the readiness and save test fixes
varunj-msft Nov 11, 2026
22b057c
Merge the readiness and save test fixes
varunj-msft Nov 11, 2026
374c620
Merge main
varunj-msft Nov 11, 2026
7ac059c
Answer suite readiness storage failures through the shared storage ha…
varunj-msft Nov 11, 2026
e08b67a
Merge main
varunj-msft Nov 11, 2026
7fa4ca4
Expect storage errors from main's version-checked delete tests the wa…
varunj-msft Nov 11, 2026
9bf5886
Merge main
varunj-msft Nov 11, 2026
626d2d4
Merge main
varunj-msft Nov 11, 2026
4a86017
Merge main
varunj-msft Nov 11, 2026
9568edb
Merge main
varunj-msft Nov 11, 2026
54e4374
Merge main
varunj-msft Nov 11, 2026
0d97131
Merge main
varunj-msft Nov 11, 2026
2849ae8
Merge main
varunj-msft Nov 11, 2026
e34258c
Fit the explicit-target E2E from main to the dialog's save button and…
varunj-msft Nov 11, 2026
bf91430
Merge the explicit-target E2E fix
varunj-msft Nov 11, 2026
a9dccb3
Merge the explicit-target E2E fix
varunj-msft Nov 11, 2026
c506195
Merge main
varunj-msft Nov 11, 2026
1ba526f
Merge main
varunj-msft Nov 11, 2026
9cdf7e7
Merge main
varunj-msft Nov 11, 2026
40d36ed
Merge main
varunj-msft Nov 11, 2026
6a52054
Merge main
varunj-msft Nov 11, 2026
23a7ec1
Merge main
varunj-msft Nov 11, 2026
70fbc3d
Merge main
varunj-msft Nov 11, 2026
a1e79d1
Merge main
varunj-msft Nov 11, 2026
a5d42d1
Merge main
varunj-msft Nov 11, 2026
2402df2
Merge main
varunj-msft Nov 11, 2026
8c517d0
Use the dialog's save button in main's endpoint tests
varunj-msft Nov 11, 2026
1137977
Merge the endpoint test fix
varunj-msft Nov 11, 2026
aaefa8f
Merge the endpoint test fix
varunj-msft Nov 11, 2026
0e7a21f
Exempt the storage error handler from the keyword-only check
varunj-msft Nov 11, 2026
891e8d6
Make get_target's parameters keyword-only
varunj-msft Nov 11, 2026
c35fa01
Merge keyword-only fixes
varunj-msft Nov 11, 2026
953dcce
Merge keyword-only fixes
varunj-msft Nov 11, 2026
d6079a1
Merge keyword-only fixes
varunj-msft Nov 11, 2026
21556ed
Merge keyword-only fixes
varunj-msft Nov 11, 2026
14004e9
Merge keyword-only fixes
varunj-msft Nov 11, 2026
3abc824
Merge keyword-only fixes
varunj-msft Nov 11, 2026
623fd40
Merge keyword-only fixes
varunj-msft Nov 11, 2026
2d74284
Merge keyword-only fixes
varunj-msft Nov 11, 2026
1fc386a
Merge keyword-only fixes
varunj-msft Nov 11, 2026
bf97225
Log every problem with a saved file that isn't valid
varunj-msft Nov 11, 2026
5e6faed
Merge the full schema-validation log
varunj-msft Nov 11, 2026
7625f89
Merge the full schema-validation log
varunj-msft Nov 11, 2026
8dbd4c2
Merge the full schema-validation log
varunj-msft Nov 11, 2026
5c7223a
Merge the full schema-validation log
varunj-msft Nov 11, 2026
8676ee5
Merge the full schema-validation log
varunj-msft Nov 11, 2026
74e1310
Merge the full schema-validation log
varunj-msft Nov 11, 2026
35199cc
Leave the error behind a storage failure out of the logs
varunj-msft Nov 12, 2026
efcdfa9
Test a blob upload that fails with a transport or authentication error
varunj-msft Nov 12, 2026
0fdebdc
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
c413994
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
c057e30
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
dc97ec7
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
a9478df
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
dc0c058
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
8f27c91
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
4f2208d
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
d22d466
Merge the storage log and restore report fixes
varunj-msft Nov 12, 2026
c9b795a
Merge main
varunj-msft Nov 12, 2026
38c981c
Merge main
varunj-msft Nov 12, 2026
eb71448
Merge main
varunj-msft Nov 12, 2026
1be067e
Merge main
varunj-msft Nov 12, 2026
162fc2a
Merge main
varunj-msft Nov 12, 2026
6c0c3dd
Merge main
varunj-msft Nov 12, 2026
3d0950d
Merge main
varunj-msft Nov 12, 2026
8b59c5d
Merge main
varunj-msft Nov 12, 2026
d1504ec
Merge main
varunj-msft Nov 12, 2026
457563c
Merge main
varunj-msft Nov 12, 2026
4e8d9c2
Return 409 when a suite's launch scripts would replace a saved entry
varunj-msft Nov 12, 2026
b1bbbe6
Merge the suite launch conflict status
varunj-msft Nov 12, 2026
9ce29cb
Merge the suite launch conflict status
varunj-msft Nov 12, 2026
4e0e640
Merge the suite launch conflict status
varunj-msft Nov 12, 2026
0b9efb3
Merge the suite launch conflict status
varunj-msft Nov 12, 2026
28c9398
Merge the suite launch conflict status
varunj-msft Nov 12, 2026
15e00de
Merge main
varunj-msft Nov 12, 2026
0c5baa8
Merge main
varunj-msft Nov 12, 2026
a3f65ad
Merge main
varunj-msft Nov 12, 2026
dd7358b
Merge main
varunj-msft Nov 12, 2026
fc37262
Merge main
varunj-msft Nov 12, 2026
d4337d8
Merge main
varunj-msft Nov 12, 2026
48bc6d1
Merge main
varunj-msft Nov 12, 2026
9789264
Merge main
varunj-msft Nov 12, 2026
42b9853
Merge main
varunj-msft Nov 12, 2026
fd7a648
Merge main
varunj-msft Nov 12, 2026
805ca2a
Merge main into part 7
varunj-msft Nov 13, 2026
b9c50e1
Merge part 7 into part 8
varunj-msft Nov 13, 2026
5363c0b
Merge part 8 into part 9
varunj-msft Nov 13, 2026
d50f393
Merge part 9 into part 10
varunj-msft Nov 13, 2026
1c80cf5
Merge part 10 into part 11
varunj-msft Nov 13, 2026
cdbe6ad
Keep the recipe storage setup apart from the saved target requirements
varunj-msft Nov 13, 2026
450b1e5
Merge part 7 into part 8
varunj-msft Nov 13, 2026
699ab5b
Move saving a target into the registry library
varunj-msft Nov 13, 2026
7d914b1
Merge part 8 into part 9
varunj-msft Nov 13, 2026
b008ce5
Merge part 9 into part 10
varunj-msft Nov 13, 2026
47ba126
Merge part 10 into part 11
varunj-msft Nov 13, 2026
713f7d1
Merge main into part 8
varunj-msft Nov 13, 2026
793316a
Merge part 8 into part 9
varunj-msft Nov 13, 2026
4ca4f25
Merge part 9 into part 10
varunj-msft Nov 13, 2026
e4f32e3
Merge part 10 into part 11
varunj-msft Nov 13, 2026
2e2e85d
Merge main into part 11
varunj-msft Nov 13, 2026
048283e
Merge main into part 8
varunj-msft Nov 13, 2026
6c243b6
Merge part 8 into part 9
varunj-msft Nov 13, 2026
1411c5d
Merge part 9 into part 10
varunj-msft Nov 13, 2026
b248a32
Merge part 10 into part 11
varunj-msft Nov 13, 2026
f273bff
Merge main into part 8
varunj-msft Nov 14, 2026
d692b25
Merge main into part 9
varunj-msft Nov 14, 2026
dc2a52c
Merge main into part 10
varunj-msft Nov 14, 2026
24bf3b2
Merge main into part 11
varunj-msft Nov 14, 2026
54d9332
Check saved entries against restore's inputs and settle cancelled saves
varunj-msft Nov 14, 2026
0529525
Merge part 8 into part 9
varunj-msft Nov 14, 2026
d0bb066
Ask the backend whether a target's values can be saved
varunj-msft Nov 14, 2026
eef542d
Merge part 9 into part 10, keeping who saved a target in capability d…
varunj-msft Nov 14, 2026
3de68f9
Merge part 10 into part 11
varunj-msft Nov 14, 2026
c05e125
Check saveability with a request that registers nothing
varunj-msft Nov 14, 2026
0d6cc7a
Merge part 9 into part 10
varunj-msft Nov 14, 2026
0310478
Merge part 10 into part 11
varunj-msft Nov 14, 2026
e47689c
Wrap the saveability note in the backend README
varunj-msft Nov 14, 2026
3a81a37
Merge part 10
varunj-msft Nov 14, 2026
b3af544
Merge part 11
varunj-msft Nov 14, 2026
e6cfbd9
Answer the saveability check in the numeric controls end-to-end mocks
varunj-msft Nov 14, 2026
192552a
Merge part 10
varunj-msft Nov 14, 2026
bd98bb1
Merge part 11
varunj-msft Nov 14, 2026
87de386
Bound the URL scheme when redacting text
varunj-msft Nov 14, 2026
96361e5
Merge part 9
varunj-msft Nov 14, 2026
e1ab38a
Merge part 10
varunj-msft Nov 14, 2026
977ad50
Merge part 11
varunj-msft Nov 14, 2026
33650ff
Merge main into part 8
varunj-msft Nov 17, 2026
87a4abf
Merge main into part 9
varunj-msft Nov 17, 2026
b369843
Merge main into part 9
varunj-msft Nov 17, 2026
00adffb
Merge main into part 10
varunj-msft Nov 17, 2026
5f1871b
Merge main into part 11
varunj-msft Nov 17, 2026
ed3141f
Merge main into part 9
varunj-msft Nov 18, 2026
f691ba6
Keep saveability available as a target name
varunj-msft Nov 18, 2026
0bba24f
Merge main into part 10
varunj-msft Nov 18, 2026
1ac1352
Merge main into part 11
varunj-msft Nov 18, 2026
2c15d96
Merge main into part 9
varunj-msft Nov 19, 2026
d055396
Keep the class save rule in the library and answer with the cause alone
varunj-msft Nov 19, 2026
ca4324b
Create targets only you can use, or that allow credential updates, as…
varunj-msft Nov 19, 2026
d331b4a
Take out who saved each target
varunj-msft Nov 19, 2026
6a22e0d
Merge part 9 into part 11
varunj-msft Nov 19, 2026
b2ad86f
Expect a class's own reason when it can't be saved
varunj-msft Nov 19, 2026
d9a66f6
Give each reason a target stays Temporary as its cause only, and drop…
varunj-msft Nov 19, 2026
8bc4161
Say the saveability check runs create's checks on the type, parameter…
varunj-msft Nov 19, 2026
9eff025
Merge part 9 into part 11
varunj-msft Nov 19, 2026
fa087b1
Merge main into part 9
varunj-msft Nov 19, 2026
0dc92aa
Merge main into part 9
varunj-msft Nov 19, 2026
d45e575
Merge part 9 into part 11
varunj-msft Nov 19, 2026
77e2514
Merge main into part 11
varunj-msft Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/getting_started/pyrit_conf.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,8 @@ A saved file never holds a credential: restore refuses a file that stores a valu
- In identity mode, an OpenAI target file sets `"headers": "{}"` in its `params` so that the server's `OPENAI_ADDITIONAL_REQUEST_HEADERS` aren't added. The OpenAI SDK adds the headers in `OPENAI_CUSTOM_HEADERS` to every request, so while the server sets it, OpenAI targets can't be saved and saved ones aren't restored. The SDK's `OPENAI_ORG_ID` and `OPENAI_PROJECT_ID` still apply to every OpenAI target on this server.
- GitHub Copilot, Hugging Face, HackAPrompt, LiteLLM, HTTP, and Copilot WebSocket targets can't be saved yet.

Converters and scorers follow the same rules. The Azure Speech converters, the Azure Content Safety scorer, and the audio scorers can't be saved yet, because they sign in with this server's key or Azure identity, and neither can the local refusal classifier scorer, whose model can come from this server.

### `initialization_scripts`

Local paths to custom Python scripts containing `PyRITInitializer` subclasses. Paths can be absolute or relative to the current working directory.
Expand Down
6 changes: 5 additions & 1 deletion pyrit/converter/azure_speech_audio_to_text_converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
import asyncio
import logging
import time
from collections.abc import Awaitable, Callable
from collections.abc import Awaitable, Callable, Mapping
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, ClassVar

if TYPE_CHECKING:
Expand Down Expand Up @@ -44,6 +45,9 @@ class AzureSpeechAudioToTextConverter(Converter):
#: The resource ID for accessing the service when using Entra ID auth.
AZURE_SPEECH_RESOURCE_ID_ENVIRONMENT_VARIABLE: str = "AZURE_SPEECH_RESOURCE_ID"

# An empty key reads AZURE_SPEECH_KEY, or else signs in with this server's Azure identity.
server_fallback_parameters: ClassVar[Mapping[str, str | None]] = MappingProxyType({"azure_speech_key": None})

COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset(
{"azure_speech_region", "azure_speech_key", "azure_speech_resource_id"}
)
Expand Down
6 changes: 5 additions & 1 deletion pyrit/converter/azure_speech_text_to_audio_converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
# Licensed under the MIT license.

import logging
from collections.abc import Awaitable, Callable
from collections.abc import Awaitable, Callable, Mapping
from types import MappingProxyType
from typing import TYPE_CHECKING, ClassVar, Literal

if TYPE_CHECKING:
Expand Down Expand Up @@ -42,6 +43,9 @@ class AzureSpeechTextToAudioConverter(Converter):
#: The resource ID for accessing the service when using Entra ID auth.
AZURE_SPEECH_RESOURCE_ID_ENVIRONMENT_VARIABLE: str = "AZURE_SPEECH_RESOURCE_ID"

# An empty key reads AZURE_SPEECH_KEY, or else signs in with this server's Azure identity.
server_fallback_parameters: ClassVar[Mapping[str, str | None]] = MappingProxyType({"azure_speech_key": None})

COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset(
{"azure_speech_region", "azure_speech_key", "azure_speech_resource_id"}
)
Expand Down
9 changes: 8 additions & 1 deletion pyrit/converter/converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import inspect
import re
from dataclasses import dataclass
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, ClassVar, cast, get_args

from pyrit import converter
Expand All @@ -18,7 +19,7 @@

if TYPE_CHECKING:
import random
from collections.abc import Awaitable, Callable
from collections.abc import Awaitable, Callable, Mapping

from pyrit.prompt_target import PromptTarget

Expand Down Expand Up @@ -73,6 +74,12 @@ class Converter(Identifiable):
#: ``super().__init__(converter_target=...)`` so the base class can validate it.
TARGET_REQUIREMENTS: ClassVar[TargetRequirements] = TargetRequirements()

# Credential parameters that, left empty, let the converter fall back to a credential from this
# server (an environment variable, or the server's Azure identity), each mapped to the value that
# turns that off, or None when no value does. A saved converter never falls back to this server's
# credentials, so a saved entry that would is refused.
server_fallback_parameters: ClassVar[Mapping[str, str | None]] = MappingProxyType({})

_identifier: ComponentIdentifier | None = None
_seed: int | None = None

Expand Down
7 changes: 3 additions & 4 deletions pyrit/registry/registry_recipe_restore.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,11 @@ class RegistryRecipeRestorer:
way the caller's create builds one; the backend's builder calls the API's create for each kind. A
file never chooses what gets imported: classes and attack types are looked up among the ones already
registered. Server paths are refused as they are for API input, and ``check_saved_entry_params``
refuses a recipe that stores a credential, or a target recipe that would let its target use one from
this server or send requests where this server's settings say. Entries that can't come back are
refuses a recipe that stores a credential, that would let its entry use one from this server, or, for
a target, that would send requests where this server's settings say. Entries that can't come back are
reported with the reason instead of being skipped.

Stored files are otherwise trusted the way configuration is. Converters and scorers don't
declare the credentials they read from server settings yet, so those rules cover targets only.
Stored files are otherwise trusted the way configuration is.
"""

_KIND_BY_COMPONENT_TYPE: ClassVar[dict[ComponentType, RegistryRecipeKind]] = {
Expand Down
8 changes: 3 additions & 5 deletions pyrit/registry/registry_recipe_rules.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,11 +168,9 @@ def _check_server_fallbacks(*, cls: type, params: Mapping[str, Any], auth_mode:
"""
if auth_mode is None:
auth_mode = "api_key"
supported_auth_modes = getattr(cls, "supported_auth_modes", None)
if supported_auth_modes is not None and not supported_auth_modes:
# A class that declares no auth modes, such as TextTarget, takes no credentials of its own.
supported_auth_modes = ("api_key",)
if supported_auth_modes is not None and auth_mode not in supported_auth_modes:
# A class that declares no auth modes takes none, as check_saveable_class assumes.
supported_auth_modes = getattr(cls, "supported_auth_modes", None) or ("api_key",)
if auth_mode not in supported_auth_modes:
modes = ", ".join(f"'{mode}'" for mode in supported_auth_modes)
raise SavedEntryRefusedError(
reason=RestoreFailureReason.UNSUPPORTED_CREDENTIAL,
Expand Down
5 changes: 5 additions & 0 deletions pyrit/score/float_scale/audio_float_scale_scorer.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

from typing import ClassVar

from pyrit.models import ComponentIdentifier, MessagePiece, Score, ScoringExpectation
from pyrit.score.audio_transcript_scorer import AudioTranscriptHelper
Expand All @@ -17,6 +18,10 @@ class AudioFloatScaleScorer(MessageFloatScaleScorer):
then scores the transcript using a FloatScaleScorer.
"""

not_saveable_reason: ClassVar[str | None] = (
"It transcribes audio with the Azure Speech key or Azure identity this server is set up with."
)

_default_validator: ScorerPromptValidator = ScorerPromptValidator(supported_data_types=["audio_path"])

def __init__(
Expand Down
8 changes: 6 additions & 2 deletions pyrit/score/float_scale/azure_content_filter_scorer.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@

import base64
import logging
from collections.abc import Awaitable, Callable # noqa: TC003 - registry annotation resolution
from typing import TYPE_CHECKING
from collections.abc import Awaitable, Callable, Mapping # noqa: TC003 - registry annotation resolution
from types import MappingProxyType
from typing import TYPE_CHECKING, ClassVar

from azure.ai.contentsafety.aio import ContentSafetyClient
from azure.ai.contentsafety.models import (
Expand Down Expand Up @@ -57,6 +58,9 @@ class AzureContentFilterScorer(MessageFloatScaleScorer):
API_KEY_ENVIRONMENT_VARIABLE: str = "AZURE_CONTENT_SAFETY_API_KEY"
ENDPOINT_URI_ENVIRONMENT_VARIABLE: str = "AZURE_CONTENT_SAFETY_API_ENDPOINT"

# An empty key reads AZURE_CONTENT_SAFETY_API_KEY, or else signs in with this server's Azure identity.
server_fallback_parameters: ClassVar[Mapping[str, str | None]] = MappingProxyType({"api_key": None})

# Mapping from Azure TextCategory to evaluation file configurations
_CATEGORY_EVAL_FILES: dict[TextCategory, tuple[list[str], str, str]] = {
TextCategory.HATE: (["harm/hate_speech.csv"], "harm/hate_speech_metrics.jsonl", "hate_speech"),
Expand Down
12 changes: 11 additions & 1 deletion pyrit/score/scorer.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from abc import abstractmethod
from dataclasses import replace
from datetime import UTC, datetime
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, ClassVar, TypeVar, final, overload

from pyrit.common import print_deprecation_message
Expand Down Expand Up @@ -53,7 +54,7 @@
)

if TYPE_CHECKING:
from collections.abc import Awaitable, Callable, Sequence
from collections.abc import Awaitable, Callable, Mapping, Sequence

from pyrit.exceptions import ComponentRole
from pyrit.models import ChatMessageRole
Expand Down Expand Up @@ -294,6 +295,15 @@ class Scorer(Identifiable, abc.ABC):
CONDITION_TYPE: ClassVar[type[Condition] | None] = None
SUPPORTED_SCORABLE_TYPES: ClassVar[tuple[type[Scorable], ...]] = ()

# Credential parameters that, left empty, let the scorer fall back to a credential from this
# server (an environment variable, or the server's Azure identity), each mapped to the value that
# turns that off, or None when no value does. A saved scorer never falls back to this server's
# credentials, so a saved entry that would is refused.
server_fallback_parameters: ClassVar[Mapping[str, str | None]] = MappingProxyType({})

# Why this scorer type can't be saved even without a credential, or None when it can.
not_saveable_reason: ClassVar[str | None] = None

_identifier: ComponentIdentifier | None = None

def __init_subclass__(cls, **kwargs: Any) -> None:
Expand Down
5 changes: 5 additions & 0 deletions pyrit/score/true_false/audio_true_false_scorer.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

from typing import ClassVar

from pyrit.models import ComponentIdentifier, MessagePiece, Score, ScoringExpectation
from pyrit.score.audio_transcript_scorer import AudioTranscriptHelper
Expand All @@ -17,6 +18,10 @@ class AudioTrueFalseScorer(MessageTrueFalseScorer):
then scores the transcript using a TrueFalseScorer.
"""

not_saveable_reason: ClassVar[str | None] = (
"It transcribes audio with the Azure Speech key or Azure identity this server is set up with."
)

_DEFAULT_VALIDATOR: ScorerPromptValidator = ScorerPromptValidator(supported_data_types=["audio_path"])

def __init__(
Expand Down
3 changes: 3 additions & 0 deletions pyrit/score/true_false/local_refusal_classifier_scorer.py
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,9 @@ class LocalRefusalClassifierScorer(MessageTrueFalseScorer):
_CATEGORY: ClassVar[str] = "refusal"
_RECIPE_VERSION: ClassVar[int] = 4
_DEFAULT_VALIDATOR: ClassVar[ScorerPromptValidator] = ScorerPromptValidator(supported_data_types=["text"])
not_saveable_reason: ClassVar[str | None] = (
"Its model can come from a folder on this server, or be downloaded with this server's Hugging Face sign-in."
)

def __init__(
self,
Expand Down
75 changes: 75 additions & 0 deletions tests/unit/registry/test_registry_recipe_restore.py
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,81 @@ async def test_refuses_a_target_that_signs_in_with_this_servers_account(self, tm
assert (failure.name, failure.reason) == ("copilot", RestoreFailureReason.UNSUPPORTED_CLASS)
assert "Copilot account" in failure.message

@pytest.mark.parametrize(
("class_module", "class_name"),
[
("pyrit.converter.azure_speech_text_to_audio_converter", "AzureSpeechTextToAudioConverter"),
("pyrit.converter.azure_speech_audio_to_text_converter", "AzureSpeechAudioToTextConverter"),
],
)
async def test_refuses_a_speech_converter_that_would_use_this_servers_key(
self, tmp_path: Path, class_module: str, class_name: str
) -> None:
recipe = {"class_module": class_module, "class_name": class_name, "params": {"azure_speech_region": "eastus"}}
_save(tmp_path, RegistryRecipeKind.CONVERTERS, "speech", recipe)

with patch.dict(os.environ, {"AZURE_SPEECH_KEY": "server-speech-key"}):
report = await _restore(tmp_path)

(failure,) = report.failures
assert (failure.name, failure.reason) == ("speech", RestoreFailureReason.UNSUPPORTED_CREDENTIAL)
assert (
failure.message
== f"'azure_speech_key' is empty, so '{class_name}' can use this server's credential instead."
)
assert "speech" not in ConverterRegistry.get_registry_singleton().instances.get_names()

async def test_refuses_a_content_filter_scorer_that_would_use_this_servers_key(self, tmp_path: Path) -> None:
recipe = {
"class_module": "pyrit.score.float_scale.azure_content_filter_scorer",
"class_name": "AzureContentFilterScorer",
"params": {"endpoint": "https://contoso.cognitiveservices.azure.com/"},
}
_save(tmp_path, RegistryRecipeKind.SCORERS, "content_safety", recipe)

with patch.dict(os.environ, {"AZURE_CONTENT_SAFETY_API_KEY": "server-safety-key"}):
report = await _restore(tmp_path)

(failure,) = report.failures
assert (failure.name, failure.reason) == ("content_safety", RestoreFailureReason.UNSUPPORTED_CREDENTIAL)
assert failure.message == (
"'api_key' is empty, so 'AzureContentFilterScorer' can use this server's credential instead."
)
assert "content_safety" not in ScorerRegistry.get_registry_singleton().instances.get_names()

async def test_refuses_an_audio_scorer_that_transcribes_with_this_servers_speech_settings(
self, tmp_path: Path
) -> None:
_save(
tmp_path,
RegistryRecipeKind.SCORERS,
"leak_substring",
{
"class_module": "pyrit.score.true_false.substring_scorer",
"class_name": "SubStringScorer",
"params": {"substring": "secret", "categories": ["leak"]},
},
)
_save(
tmp_path,
RegistryRecipeKind.SCORERS,
"audio_leak",
{
"class_module": "pyrit.score.true_false.audio_true_false_scorer",
"class_name": "AudioTrueFalseScorer",
"params": {"text_capable_scorer": "leak_substring"},
},
)

report = await _restore(tmp_path)

(failure,) = report.failures
assert (failure.name, failure.reason) == ("audio_leak", RestoreFailureReason.UNSUPPORTED_CLASS)
assert failure.message == (
"It transcribes audio with the Azure Speech key or Azure identity this server is set up with."
)
assert [ref.name for ref in report.restored] == ["leak_substring"]

async def test_restores_a_class_whose_optional_credentials_are_left_empty(self, tmp_path: Path) -> None:
TargetRegistry.get_registry_singleton().register_class(_OptionalKeyTarget)
_save(
Expand Down
Loading
Loading