Skip to content

[FIX] Keep execution-distinct seeds when deduplicating - #3773

Merged
hannahwestra25 merged 8 commits into
microsoft:mainfrom
feiiiiii5:fix/seed-dedup-identity
Nov 6, 2026
Merged

hannahwestra25 merged 8 commits into
microsoft:mainfrom
feiiiiii5:fix/seed-dedup-identity

Conversation

@feiiiiii5

@feiiiiii5 Chen Yufeiyang (feiiiiii5) commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3738.

Description

Seed deduplication could silently drop execution-distinct standalone seeds and singleton groups: for example, a prompt and an objective with the same text, different response schemas or roles, or a stored singleton objective re-added with edited conditions.

Compare standalone seeds and singleton groups by their execution identity, and compare retained multi-seed groups by complete membership. A changed stored singleton is copied into a new row and group, preserving the original. Exact duplicates and metadata-only edits still deduplicate.

Tests and Documentation

The new SQLite regression fails on the previous PR head (f9c25ff): editing a stored objective from the Paris condition to Lyon leaves only one group. It passes after restricting behavioral group retention to multi-seed groups, and verifies both persisted conditions and independent row/group IDs.

The deduplication, seed-condition and seed-prompt test files pass together: 195 passed. Ruff lint/format and targeted ty checks pass. Updated 8_seed_database.py and its paired notebook with identical deduplication rules; the 11 paired cells match, and notebook outputs and metadata are preserved.

Validation scope
python -m pytest \
  tests/unit/memory/memory_interface/test_interface_seed_dedup_identity.py \
  tests/unit/memory/test_seed_conditions.py \
  tests/unit/memory/memory_interface/test_interface_seed_prompts.py -q

Tests use the repository's SQLite fixture. The full repository suite, Azure SQL and notebook execution were not run in this update.

Standalone seeds and singleton groups were deduplicated by the behavioral key alone, so seeds that differed only by seed type, data type, or response_json_schema were silently skipped even though they execute differently. Standalone identity now includes those fields, and singleton/already-stored groups use the full member identity that multi-seed groups already use. Exact duplicates stay skipped; multi-seed group behavior is unchanged.

Fixes microsoft#3738
@hannahwestra25 hannahwestra25 self-assigned this Nov 2, 2026
Comment thread pyrit/memory/memory_interface.py Resolved
Comment thread pyrit/memory/memory_interface.py Resolved
Comment thread pyrit/memory/memory_interface.py Resolved
Comment thread pyrit/memory/memory_interface.py Resolved
Address review feedback: non-retained seeds (standalone and singleton) share one standalone execution key including role; the dual member/standalone key sets and the incoming-order dependency are gone; stored keys are computed once and only when needed. Add schema/role and metadata-only dedup regressions. tests/unit/memory: 1927 passed (one pre-existing SQLite index-plan failure unrelated to this change); tests/unit/datasets: 5280 passed; ruff clean.
Copilot AI balanced review requested due to automatic review settings November 3, 2026 01:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Singleton prompt copies with a changed group and response schema can reuse an existing seed primary key, causing an IntegrityError and rolling back the batch.

1 open issue
What changed in this PR

Addresses #3738 by refining PyRIT’s seed deduplication so execution-distinct seeds are preserved while equivalent seeds remain deduplicated.

Changes:

  • Includes seed type, data type, role, response schema, and behavioral inputs in standalone and singleton execution identity.
  • Reuses stored member identities for complete-group comparisons and documents the deduplication rules.
  • Adds seven regression tests covering distinct identities, exact duplicates, and metadata-only changes across grouped and standalone seeds.
File Description
tests/​unit/​memory/​memory_interface/​test_interface_seed_dedup_identity.py Adds regression coverage for execution identity and duplicate handling.
pyrit/​memory/​memory_interface.py Refines deduplication keys, reuses group-member identities, and updates the method docstring.

Review effort: Balanced

Comment thread pyrit/memory/memory_interface.py Resolved
A stored seed whose execution identity changes (e.g. response schema) is inserted as a new row, but it kept its old primary key, so the insert raised IntegrityError and rolled back the batch. Look up stored ids for every incoming seed (not only retained groups) and regenerate the row id for a non-retained copy before insertion, updating the caller-visible id only after the insert succeeds. Add order-symmetry and singleton-copy regressions.
Comment thread pyrit/memory/memory_interface.py Resolved
A stored grouped seed that is copied back after an execution edit keeps its original prompt_group_id, so the copy silently joins the stored seed group and a schema or role edit corrupts that group. Generate a new group id for such a copy before insert, publish it back to the caller-visible prompt after insert succeeds, and add role/response-schema regressions that assert the original group stays intact.
Comment thread pyrit/memory/memory_interface.py Resolved
Comment thread pyrit/memory/memory_interface.py Resolved
Comment thread pyrit/memory/memory_interface.py Resolved
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

Removed the redundant behavior-group retention in e06ad24e: the full member identity now decides which groups are retained, and _get_existing_seed_keys returns only the matched seeds. The unused helper, constant, and import are gone. The current head has passed the hosted Python 3.11–3.14 unit-test matrix on Linux, macOS, and Windows, plus the three pre-commit jobs.

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for contributing :)

@hannahwestra25
hannahwestra25 added this pull request to the merge queue Nov 6, 2026
Merged via the queue into microsoft:main with commit fbd4497 Nov 6, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seed deduplication silently drops distinct seed types and singleton response schemas

3 participants