Skip to content

FIX: Combine scenario group counts for unmatched attempts (#3699) - #3780

Open
Hrushikesh Ramilla (Hrushikesh-ramilla) wants to merge 1 commit into
microsoft:mainfrom
Hrushikesh-ramilla:fix/scenario-group-counts-unmatched-attempts-3699
Open

Hrushikesh Ramilla (Hrushikesh-ramilla) wants to merge 1 commit into
microsoft:mainfrom
Hrushikesh-ramilla:fix/scenario-group-counts-unmatched-attempts-3699

Conversation

@Hrushikesh-ramilla

Copy link
Copy Markdown
Contributor

Description

Fixes #3699.

When querying scenario history aggregates (get_scenario_history_aggregates_async in pyrit/memory/memory_interface.py), per-group attempt counts from group_rows are grouped by (scenario_result_id, unit_group_id) in SQL.

For unmatched attempts, raw_gid is formatted as name\x1feval_hash, which lines 7336-7338 convert to the canonical configuration hash canonical_gid = config_hash({"atomic_attack_name": name, "technique_eval_hash": eval_hash}).

Previously, run_groups[canonical_gid] = counts unconditionally assigned the row counts. If an unplanned attempt shared the same atomic attack name and technique configuration as a planned group, its row overwrote the existing canonical group's counts instead of combining the disjoint attempt populations. As a result, the group's planned completed/successful counts were overwritten, while overall run counts remained correct.

Fix

  • In MemoryInterface.get_scenario_history_aggregates_async:
    • If canonical_gid already exists in run_groups, combine the counts using combine_execution_counts([run_groups[canonical_gid], counts]).
    • Similarly, if canonical_gid != raw_gid, combine into run_groups[raw_gid] if already present.

Tests

  • Added unit regression test test_scenario_history_aggregates_combine_unplanned_attempt_counts_with_canonical_group in tests/unit/memory/memory_interface/test_interface_scenario_history.py using canonical configuration hashes and both planned and unplanned attempt records.
  • Verification:
    • pytest tests/unit/memory/memory_interface/test_interface_scenario_history.py: 25 passed.
    • pytest tests/unit/analytics/test_scenario_statistics_parity.py: 37 passed.
    • ruff check pyrit/memory/memory_interface.py tests/unit/memory/memory_interface/test_interface_scenario_history.py: 0 errors.
    • ruff format --check pyrit/memory/memory_interface.py tests/unit/memory/memory_interface/test_interface_scenario_history.py: Clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: Canonical scenario group counts are overwritten by unmatched attempts

1 participant