Skip to content

MAINT: Enable the ty unsound-assignment rule in stages (267 findings) #4402

Open

Description

@romanlutz

ty's unsound-assignment rule is switched off in pyproject.toml (unsound-assignment = "ignore" under [tool.ty.rules], where everything else runs at all = "error"). This issue tracks turning it on.

What the rule catches

It flags a variable annotated with a concrete type that is assigned a value the checker only knows as Any:

# pyrit/common/cli_helpers.py: getattr returns Any, so nothing checks that this is an int
level_value: int = getattr(logging, level_upper)

The annotation states a type that nothing verified. If the value is something else, type checking still passes and the failure shows up later at runtime, far from where the wrong value came in. With the rule on, every such annotation becomes a checked boundary.

Current state

Measured at fdeb9c5 with ty 0.0.99 on Python 3.14: 267 findings in 45 files.

uv run --frozen --extra all ty check pyrit --error unsound-assignment --output-format concise
Group Findings Files What they are
pyrit/memory/memory_models.py 196 1 The standard SQLAlchemy declarative idiom: 155 x col: Mapped[T] = mapped_column(...) and 41 x rel: Mapped[T] = relationship(...). One root cause, not 196 separate problems.
Everything else 71 44 Individual Any values (JSON, getattr, config and environment lookups, untyped third-party returns) assigned to annotated variables.

One PR or several? Several.

  1. There are two different kinds of work here. memory_models.py is a tooling decision: x: Mapped[T] = mapped_column(...) is how SQLAlchemy 2.0 models are meant to be written, so the right answer is one scoped override, not rewriting 196 declarations in the file our migrations depend on. The other 71 findings each need a local judgment about whether the value really has the annotated type, and some of them may turn out to be real unchecked assumptions.
  2. Reviewability. The 71 findings are spread over 44 files in 14 packages. A fix that adds validation or narrowing changes behavior on bad input and needs a test next to the code. That cannot be reviewed carefully in one sweep.
  3. Merge conflicts. Small per-package PRs rebase easily against the many PRs that are open at any time; one PR touching 45 files does not.
  4. No backsliding while the work is in progress. Step 1 below turns the rule on immediately, so new code is checked from day one and the remaining findings can only go down.

Plan

Step 1 (UA-0), one PR, must merge before anything else starts. Configuration only, no source changes:

  • Delete unsound-assignment = "ignore".
  • Add a permanent [[tool.ty.overrides]] entry for pyrit/memory/memory_models.py that ignores the rule, with a comment explaining the SQLAlchemy idiom.
  • Add a second, temporary [[tool.ty.overrides]] entry that ignores the rule for the 44 remaining files listed below, with a comment pointing to this issue.
  • Result: ty check pyrit is clean and every file not on the temporary list is checked.

Step 2, in parallel, one PR per chunk. Fix the findings in the chunk and remove its files from the temporary override.

ID Scope Findings
UA-1 pyrit/executor/promptgen/gcg/attack/base/attack_manager.py 12
UA-2 rest of pyrit/executor, plus pyrit/scenario, pyrit/converter, pyrit/prompt_normalizer 7
UA-3 pyrit/prompt_target, pyrit/embedding, pyrit/auth 13
UA-4 pyrit/backend 9
UA-5 pyrit/score 7
UA-6 pyrit/models, pyrit/registry 11
UA-7 pyrit/memory (without memory_models.py), pyrit/cli, pyrit/common 12

Step 3. Whoever empties the temporary override deletes it, and this issue can be closed.

How to fix a finding

In order of preference:

  1. Fix the type at the source. If the Any comes from one of our own functions, give that function a real return type.
  2. Check the value when it comes from outside the program (JSON, HTTP responses, environment, files): narrow with isinstance or validate, and raise a clear error otherwise. Add a test for the rejected case.
  3. cast(...) only when the type is guaranteed by construction, with a short comment saying why.

Do not add # type: ignore, and do not weaken the annotation to Any to silence the rule. If a finding turns out to be a real bug, say so in the PR description.

The 44 files outside memory_models.py, with finding counts
  • pyrit/auth/browser_session_copilot_authenticator.py (1)
  • pyrit/backend/middleware/runtime.py (1)
  • pyrit/backend/routes/targets.py (1)
  • pyrit/backend/routes/version.py (3)
  • pyrit/backend/services/attack_service.py (2)
  • pyrit/backend/services/finding_export.py (1)
  • pyrit/backend/services/scenario_service.py (1)
  • pyrit/cli/api_client.py (2)
  • pyrit/cli/pyrit_scan.py (2)
  • pyrit/common/cli_helpers.py (1)
  • pyrit/common/yaml_loadable.py (1)
  • pyrit/converter/decomposition_converter.py (1)
  • pyrit/embedding/openai_text_embedding.py (2)
  • pyrit/executor/attack/multi_turn/tree_of_attacks.py (1)
  • pyrit/executor/attack/single_turn/many_shot_jailbreak.py (1)
  • pyrit/executor/promptgen/gcg/attack/base/attack_manager.py (12)
  • pyrit/memory/alembic/env.py (1)
  • pyrit/memory/attack_analytics.py (1)
  • pyrit/memory/azure_sql_memory.py (2)
  • pyrit/memory/memory_interface.py (2)
  • pyrit/models/harm_category.py (2)
  • pyrit/models/messages/conversations.py (1)
  • pyrit/models/parameter.py (1)
  • pyrit/models/score/observation.py (1)
  • pyrit/models/score/scorable.py (1)
  • pyrit/prompt_normalizer/json_retry.py (1)
  • pyrit/prompt_normalizer/prompt_normalizer.py (1)
  • pyrit/prompt_target/azure_blob_storage_target.py (1)
  • pyrit/prompt_target/common/discover_target_capabilities.py (1)
  • pyrit/prompt_target/openai/openai_error_handling.py (3)
  • pyrit/prompt_target/openai/openai_response_target.py (1)
  • pyrit/prompt_target/prompt_shield_target.py (4)
  • pyrit/registry/components/target_registry.py (1)
  • pyrit/registry/registry_recipe_restore.py (1)
  • pyrit/registry/registry_recipe_rules.py (2)
  • pyrit/registry/registry_recipe_save.py (1)
  • pyrit/scenario/core/attack_technique_factory.py (1)
  • pyrit/scenario/scenarios/foundry/assert_policy.py (1)
  • pyrit/score/float_scale/local_violence_classifier_scorer.py (1)
  • pyrit/score/llm_scoring.py (1)
  • pyrit/score/observation/otel_execution_trace_source.py (1)
  • pyrit/score/scorer.py (2)
  • pyrit/score/scorer_prompt_validator.py (1)
  • pyrit/score/true_false/prompt_shield_scorer.py (1)

Status — who's doing what

(Maintained automatically; last updated 2026-11-20. Claim rules are below — first comment wins, no confirmation needed.)

ID Scope Claimed by Since PR State
UA-0 configuration (gates all others) Mohammad Saeed (@mss06s) 2026-11-18 #4407 open
UA-1 gcg/attack/base/attack_manager.py — available
UA-2 rest of executor, scenario, converter, prompt_normalizer — available
UA-3 prompt_target, embedding, auth Mehreen Munir (@clossal) 2026-11-19 waiting on UA-0
UA-4 backend SteeveCrypto (@Steeve-Crypto) 2026-11-20 waiting on UA-0
UA-5 score a12san (@sainimal1ba-hue) 2026-11-20 waiting on UA-0
UA-6 models, registry Shawn Russ (@shawnrruss) 2026-11-19 waiting on UA-0
UA-7 memory (minus memory_models.py), cli, common — available

How to contribute

  • Claim one chunk at a time, by its ID. Before you start, comment on this issue with the ID you are taking (for example UA-4).
  • First comment wins. If yours is the first comment claiming that ID, go ahead and open a PR; you do not need to wait for a reply. If someone has already claimed it, pick a different ID.
  • UA-0 goes first. UA-1 to UA-7 can be claimed at any time, but their PRs can only be opened once the UA-0 PR is merged.
  • Link this issue and the ID in your PR description.

Activity

  1. mss06s commented on Nov 18, 2026

    @mss06s

    I’ll take UA-0.

  2. clossal commented on Nov 19, 2026

    @clossal
    Contributor

    Hi, I'd like to take UA-3 (pyrit/prompt_target, pyrit/embedding, and pyrit/auth) if it's still available.

    I've reproduced all 13 unsound-assignment findings locally and started investigating the affected code and existing tests.

    Could you assign UA-3 to me? I'll keep the changes focused on the assigned scope and add regression tests where appropriate.

  3. shawnrruss commented on Nov 19, 2026

    @shawnrruss
    Contributor

    I’ll take UA-6 (pyrit/models and pyrit/registry). I’ll keep the changes scoped to that chunk and wait for UA-0 to merge before opening a PR.

  4. Steeve-Crypto commented on Nov 20, 2026

    @Steeve-Crypto
    Contributor

    I'll take UA-4 (pyrit/backend). I'll keep it scoped to that chunk and wait for UA-0 to merge before opening a PR.

  5. sainimal1ba-hue commented on Nov 20, 2026

    @sainimal1ba-hue

    I will take UA-5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions