Repository navigation
[FIX] Keep execution-distinct seeds when deduplicating - #3773
Conversation
Standalone seeds and singleton groups were deduplicated by the behavioral key alone, so seeds that differed only by seed type, data type, or response_json_schema were silently skipped even though they execute differently. Standalone identity now includes those fields, and singleton/already-stored groups use the full member identity that multi-seed groups already use. Exact duplicates stay skipped; multi-seed group behavior is unchanged. Fixes microsoft#3738
Address review feedback: non-retained seeds (standalone and singleton) share one standalone execution key including role; the dual member/standalone key sets and the incoming-order dependency are gone; stored keys are computed once and only when needed. Add schema/role and metadata-only dedup regressions. tests/unit/memory: 1927 passed (one pre-existing SQLite index-plan failure unrelated to this change); tests/unit/datasets: 5280 passed; ruff clean.
There was a problem hiding this comment.
🟡 Changes recommended
Singleton prompt copies with a changed group and response schema can reuse an existing seed primary key, causing an IntegrityError and rolling back the batch.
1 open issue
What changed in this PR
Addresses #3738 by refining PyRIT’s seed deduplication so execution-distinct seeds are preserved while equivalent seeds remain deduplicated.
Changes:
- Includes seed type, data type, role, response schema, and behavioral inputs in standalone and singleton execution identity.
- Reuses stored member identities for complete-group comparisons and documents the deduplication rules.
- Adds seven regression tests covering distinct identities, exact duplicates, and metadata-only changes across grouped and standalone seeds.
| File | Description |
|---|---|
tests/unit/memory/memory_interface/test_interface_seed_dedup_identity.py |
Adds regression coverage for execution identity and duplicate handling. |
pyrit/memory/memory_interface.py |
Refines deduplication keys, reuses group-member identities, and updates the method docstring. |
Review effort: Balanced
A stored seed whose execution identity changes (e.g. response schema) is inserted as a new row, but it kept its old primary key, so the insert raised IntegrityError and rolled back the batch. Look up stored ids for every incoming seed (not only retained groups) and regenerate the row id for a non-retained copy before insertion, updating the caller-visible id only after the insert succeeds. Add order-symmetry and singleton-copy regressions.
A stored grouped seed that is copied back after an execution edit keeps its original prompt_group_id, so the copy silently joins the stored seed group and a schema or role edit corrupts that group. Generate a new group id for such a copy before insert, publish it back to the caller-visible prompt after insert succeeds, and add role/response-schema regressions that assert the original group stays intact.
|
Removed the redundant behavior-group retention in |
hannahwestra25
left a comment
There was a problem hiding this comment.
thanks for contributing :)

Fixes #3738.
Description
Seed deduplication could silently drop execution-distinct standalone seeds and singleton groups: for example, a prompt and an objective with the same text, different response schemas or roles, or a stored singleton objective re-added with edited conditions.
Compare standalone seeds and singleton groups by their execution identity, and compare retained multi-seed groups by complete membership. A changed stored singleton is copied into a new row and group, preserving the original. Exact duplicates and metadata-only edits still deduplicate.
Tests and Documentation
The new SQLite regression fails on the previous PR head (
f9c25ff): editing a stored objective from the Paris condition to Lyon leaves only one group. It passes after restricting behavioral group retention to multi-seed groups, and verifies both persisted conditions and independent row/group IDs.The deduplication, seed-condition and seed-prompt test files pass together: 195 passed. Ruff lint/format and targeted
tychecks pass. Updated8_seed_database.pyand its paired notebook with identical deduplication rules; the 11 paired cells match, and notebook outputs and metadata are preserved.Validation scope
Tests use the repository's SQLite fixture. The full repository suite, Azure SQL and notebook execution were not run in this update.