[Chen Yufeiyang (feiiiiii5)]

Fixes #3738.

Description

Seed deduplication could silently drop execution-distinct standalone seeds and singleton groups: for example, a prompt and an objective with the same text, different response schemas or roles, or a stored singleton objective re-added with edited conditions.

Compare standalone seeds and singleton groups by their execution identity, and compare retained multi-seed groups by complete membership. A changed stored singleton is copied into a new row and group, preserving the original. Exact duplicates and metadata-only edits still deduplicate.

Tests and Documentation

The new SQLite regression fails on the previous PR head (f9c25ff): editing a stored objective from the Paris condition to Lyon leaves only one group. It passes after restricting behavioral group retention to multi-seed groups, and verifies both persisted conditions and independent row/group IDs.

The deduplication, seed-condition and seed-prompt test files pass together: 195 passed. Ruff lint/format and targeted ty checks pass. Updated 8_seed_database.py and its paired notebook with identical deduplication rules; the 11 paired cells match, and notebook outputs and metadata are preserved.

Validation scope

python -m pytest \
  tests/unit/memory/memory_interface/test_interface_seed_dedup_identity.py \
  tests/unit/memory/test_seed_conditions.py \
  tests/unit/memory/memory_interface/test_interface_seed_prompts.py -q

Tests use the repository's SQLite fixture. The full repository suite, Azure SQL and notebook execution were not run in this update.

[hannahwestra25]

Role isn't in this key, so standalone user and system prompts with the same text still collapse to one row and the system prompt is lost. That's the same execution-distinct case #3738 is about. Normalize None to "user" to match SeedGroup:

Suggested change

	"data_type": entry.data_type,
	"data_type": entry.data_type,
	"role": entry.role or "user",

[Chen Yufeiyang (feiiiiii5)]

Fixed: _seed_standalone_key now includes "role": entry.role or "user", matching SeedGroup's None->user normalization, so a standalone user and system seed with the same text stay distinct. Covered by the new schema/role test.

[hannahwestra25]

Each stored seed now builds SeedEntry + JSON for the member key 2–3 times, plus a standalone key even with no standalone input. Reloading 3,000 stored seeds was ~1.6–2x slower than main. If member identity stays, compute once:

Suggested change

	stored_member_pairs = {
	(dataset, self._seed_group_member_key(seed)) for dataset, seed in existing_seeds if dataset
	}
	stored_member_keys = {self._seed_group_member_key(seed) for dataset, seed in existing_seeds if not dataset}
	member_keys = [(dataset, self._seed_group_member_key(seed)) for dataset, seed in existing_seeds]
	stored_member_pairs = {(dataset, key) for dataset, key in member_keys if dataset}
	stored_member_keys = {key for dataset, key in member_keys if not dataset}

[Chen Yufeiyang (feiiiiii5)]

Addressed: each stored seed's member key is computed once and passed to _get_new_seed_group_ids; the standalone-key sets are built in one pass and skipped entirely when no incoming seed needs them. On a 300-seed reload benchmark: main 27.0ms, old PR head 39.4ms, this change 36.6ms (the remaining cost is the execution-identity key itself).

[Copilot]

🟡 Changes recommended

Singleton prompt copies with a changed group and response schema can reuse an existing seed primary key, causing an IntegrityError and rolling back the batch.

1 open issue

• Regenerate stored seed IDs when copying singleton prompts · New

What changed in this PR

Addresses #3738 by refining PyRIT’s seed deduplication so execution-distinct seeds are preserved while equivalent seeds remain deduplicated.

Changes:

• Includes seed type, data type, role, response schema, and behavioral inputs in standalone and singleton execution identity.
• Reuses stored member identities for complete-group comparisons and documents the deduplication rules.
• Adds seven regression tests covering distinct identities, exact duplicates, and metadata-only changes across grouped and standalone seeds.

File	Description
tests/​unit/​memory/​memory_interface/​test_interface_seed_dedup_identity.py	Adds regression coverage for execution identity and duplicate handling.
pyrit/​memory/​memory_interface.py	Refines deduplication keys, reuses group-member identities, and updates the method docstring.

_{Review effort: Balanced}

[hannahwestra25]

Objectives never reach this copy path: a stored single-seed objective re-added with edited conditions is still silently dropped. _get_new_behavior_group_ids retains its one-seed group, so the 5095 skip fires. Fix: retain only multi-seed groups (lines 5287-5290, outside the diff):

        group_sizes = Counter(seed.prompt_group_id for seed in seeds)
        group_ids: set[uuid.UUID] = set()
        for seed in seeds:
            if (
                not isinstance(seed, SeedObjective)
                or seed.prompt_group_id is None
                or group_sizes[seed.prompt_group_id] < 2
                or not seed.value_sha256
            ):
                continue

Regression test (fails on head):

from pyrit.models import AnswerMatches, Seed, SeedGroup, SeedObjective, SeedPrompt


async def test_readding_objective_with_edited_conditions_keeps_both(sqlite_instance: MemoryInterface) -> None:
    """An objective re-added with edited conditions is execution-distinct, so both must stay."""

    def objective(answer: str) -> SeedObjective:
        return SeedObjective(
            value="capital?",
            dataset_name=DATASET,
            conditions=(AnswerMatches(correct_answer=answer, correct_answer_label="A"),),
        )

    await _add(sqlite_instance, *SeedGroup(seeds=[objective("Paris")]).seeds)
    stored = (await sqlite_instance.get_seeds_async(dataset_name=DATASET))[0]

    stored.conditions = objective("Lyon").conditions
    await _add(sqlite_instance, *SeedGroup(seeds=[stored]).seeds)

    groups = await sqlite_instance.get_seed_groups_async(dataset_name=DATASET)
    assert sorted(len(group.seeds) for group in groups) == [1, 1]

[Chen Yufeiyang (feiiiiii5)]

Thanks for catching the retained-singleton path. Fixed in 13d0e08: behavioral retention now applies only to multi-seed groups. The SQLite regression fails on the previous head and now verifies both singleton groups, new row/group IDs, and the unchanged Paris condition alongside the Lyon copy. The three related seed test files pass together (195 tests); Ruff and targeted ty checks pass.

[hannahwestra25]

The public docs still state the old rule. doc/code/memory/8_seed_database.py:27-30 and the paired .ipynb markdown cell are outside this diff, so this is plain text. Update both identically. Rule 2 assumes the conditions fix.

doc/code/memory/8_seed_database.py:

# PyRIT uses content hashing to avoid re-importing stored seeds. Standalone seeds and single-seed groups follow these rules:
#
# 1. **Same dataset, duplicate content and execution identity** (seed type, data type, role, response schema, and behavioral inputs such as conditions): Seed is skipped, even if descriptive metadata such as harm categories differs
# 2. **Same dataset, modified content or execution identity**: Seed is accepted as a new row; the stored original is unchanged

doc/code/memory/8_seed_database.ipynb:

    "PyRIT uses content hashing to avoid re-importing stored seeds. Standalone seeds and single-seed groups follow these rules:\n",
    "\n",
    "1. **Same dataset, duplicate content and execution identity** (seed type, data type, role, response schema, and behavioral inputs such as conditions): Seed is skipped, even if descriptive metadata such as harm categories differs\n",
    "2. **Same dataset, modified content or execution identity**: Seed is accepted as a new row; the stored original is unchanged\n",

[Chen Yufeiyang (feiiiiii5)]

Updated both 8_seed_database.py and the paired notebook in 13d0e08 with the same execution-identity and copy semantics. The 11 paired cells match; only the existing Markdown cell changed, and notebook outputs and metadata are preserved.