[Aditya Malkar (adimalkar)]

Description

Fixes #3733.

On main (f3a88daf), with in-memory SQLite, a target whose _send_prompt_to_target_async signals entry and then blocks runs PromptSendingAttack.execute_async(objective="cancellation audit probe") as a task. I cancel the task once the target is entered. CancelledError propagates, and memory holds 0 message pieces and 0 attack results. With this change the same run keeps [("user", "cancellation audit probe")] and one AttackResult with outcome UNDETERMINED, pointing at that conversation, then still propagates CancelledError.

There were two separate gaps, since CancelledError is a BaseException:

PromptNormalizer.send_prompt_async persisted the request after the target returned or in except Exception, and cancellation skipped both. It now has an except asyncio.CancelledError branch:

• The converted request is persisted, with no response because none was received, and the cancellation is re-raised.
• It uses the same rule as the Exception branch: if a send_context reports the target was never invoked, nothing is persisted.
• A failure to write the request is logged and never replaces the cancellation.

AttackStrategy.execute_with_context_async only reaches ON_ERROR (which writes the error result) for Exception. It now handles CancelledError too:

• If the attack had started sending to the objective target, it persists an AttackResult and re-raises. _ObjectiveTargetConversationLifecycle already records those invocations; I added invoked.
• The result has outcome UNDETERMINED. It is not marked success, failure or error, and has no error fields.
• It carries the same identifiers as an error result (conversation, related conversations, attribution, labels and retry events), plus a typed marker in metadata.
• That marker is a new AttackCancellation in pyrit/executor/attack/core/attack_cancellation.py, modelled on AttackPreparationFailure (to_metadata / from_result).
• A failed result write is logged, confirmed by ID like the error path, and doesn't mask the cancellation. Cancellation before any objective-target send records nothing.

Scenario resume. _get_completed_objective_hashes_async counts every non-ERROR row as completed, including UNDETERMINED. So without a change there, a resumed scenario would skip an objective that was cancelled mid-flight. I checked: with the exclusion removed, the updated Crescendo resume test fails with B's attack never run. Rows carrying AttackCancellation now stay pending, like ERROR rows. As with an ERROR row today, the cancelled attempt stays in ScenarioResult.attack_results next to the resumed one, since supersession is per conversation and the resumed attempt uses a new one.

Tests and Documentation

New tests:

• tests/unit/prompt_normalizer/test_prompt_normalizer.py:
  • A cancelled target call persists only the request.
  • A cancelled send whose send_context never reached the target persists nothing.
  • A failing request write doesn't mask CancelledError.
• tests/unit/executor/attack/single_turn/test_prompt_sending.py, cancellation at three points plus a failed write:
  • Before the target call (blocked in setup): nothing is recorded.
  • During the call: the request and an UNDETERMINED result with the marker are kept, with no error fields.
  • After the target responds (blocked in scoring): the exchange and an UNDETERMINED result are kept.
  • A failing result write doesn't mask CancelledError.

Existing tests updated, because they asserted that cancellation leaves no trace:

• test_crescendo_resilience.py::test_cancellation_runs_teardown_preserves_completed_turns_and_clears_retry_state: 9 pieces (4 turns plus the in-flight request) and one cancelled UNDETERMINED result.
• test_scenario_crescendo_resume.py::test_crescendo_scenario_cancellation_preserves_progress_and_resumes_only_incomplete_objective: the cancelled attempt at B is recorded. Resume still re-runs B, and the partial conversation now ends with the in-flight B prompt 5.
• test_github_copilot_target.py::test_normalizer_deletes_owned_session_when_creation_is_cancelled_after_allocation_async: cancellation lands inside _send_prompt_to_target_async (session allocation), so the request is persisted. The Exception path does the same in that phase. The session-cleanup assertions are unchanged.

The 5 behavioural tests fail on main and pass here. tests/unit has the same failures on main and on this branch (55 failed / 89 errors locally, all litellm-dependent or otherwise environmental), plus 7 new passing tests. ruff and ty check pyrit (lockfile versions) report nothing new.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.