Skip to content

Python 3.11 adoption opportunities: claim one improvement at a time #3166

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

PyRIT now requires Python >=3.11 (currently <3.15), and Ruff targets py311. We can use Python 3.11 features consistently across supported versions, simplify older compatibility patterns, and improve typing, diagnostics, and async lifecycle management. Some features are already partially adopted; the opportunities below are investigations, not instructions to replace every existing pattern.

Describe the solution you'd like

How to participate: ONE opportunity at a time

Claim exactly ONE checklist item at a time by commenting on this issue thread with its ID (for example, P311-01) and a short description of what you intend to investigate. Do not start multiple items or claim a bundle. Wait for confirmation here before treating the item as reserved. Claims are processed in comment order, and this issue is checked approximately every three hours.

  • You must open a PR within five days (120 hours) of your claim comment. The table below records the due date. If no PR is opened by then, the claim expires, the item becomes available, and your issue assignment is removed if you have no other active contribution on this issue. You can claim it again if it is still available.
  • Opening a PR within the deadline retains your claim while that PR is active. You cannot claim another item until your current PR is merged or the current contribution is explicitly released/closed.
  • Link this issue and the item ID in your PR description. Keep the PR focused on that one opportunity. Follow repository instructions and add relevant behavior/type-checking coverage.
  • Investigate first. Preserve public contracts and intended behavior. If a feature is not a good fit, explain why on this thread rather than forcing a change. An investigation can conclude that no code change is appropriate.
  • A PR opened without a prior claim will be highlighted to Roman Lutz (@romanlutz). If the item is unclaimed, we can link the PR and assign the item to its author. If someone else has already claimed it or another PR overlaps, Roman Lutz (@romanlutz) will assess how to proceed; no existing claim will be displaced automatically.
  • Checkboxes mean completed, not claimed. Check an item only after its PR is merged, or after a maintainer explicitly accepts a no-change investigation. Ownership and PR links are tracked separately below.

Opportunity checklist

Typing

  • P311-01: Extend typing.Self adoption. Investigate older same-concrete-class factories and fluent APIs, such as pyrit/common/yaml_loadable.py:YamlLoadable.from_yaml_file. Replace bound-TypeVar boilerplate only where it really models Self; retain independent generic parameters. Merged scope: #3194 covers YAML/seed and scorer-metrics loaders, strategy-context copying, and technique collections, with type-regression coverage. Additional merged scope: #3201 covers the three Message factories and AtomicAttackIdentifier.build, including subclass type-regression tests. Additional merged scope: #3266 covers SeedDataset.from_dict, MessageScorable.from_message, and ContentScorable.from_message, with subclass runtime and assert_type coverage and unchanged validation. Additional merged scope: #3383 covers AttackTechniqueSeedGroup.from_system_prompt and from_messages, with unchanged cls construction/validation and subclass runtime/assert_type coverage. Additional merged scope: #3411 covers MessagePiece.adversarial_placeholder with unchanged cls construction and subclass runtime/assert_type coverage. Additional merged scope: #3470 covers TagQuery.all, any_of, and none_of, with subclass runtime/assert_type coverage and unchanged predicates/validation. Additional merged scope: #3520 covers ScenarioRunSizeEstimate.unavailable, preserving cls construction, unavailable status, defaults, dataset/limit handling and validation, with subclass runtime/assert_type and input/default coverage. Additional merged scope: #3529 covers HarmDefinition.from_yaml, with subclass runtime/assert_type coverage and unchanged YAML loading, path resolution, field conversion, source-path recording and validation. Additional merged scope: #3577 annotates JsonResponseConfig.from_metadata with Self and adds subclass runtime/assert_type coverage; the factory runtime body, schema parsing, defaults, validation and metadata round trips are unchanged. Additional merged scope: #3626 annotates WebSearchConfig.from_metadata with Self, with subclass runtime/assert_type coverage of absent and explicit policies. Construction, read-only Mapping inputs, default/disabled/required modes, frozen/extra-forbid validation, errors, non-mutation and metadata/JSON round trips are unchanged. Additional merged scope: #3779 annotates AdversarialPrompts.from_metadata with Self | None and adds base/subclass runtime and assert_type checks, absent-association cases, read-only mapping, non-mutation, JSON round-trip and validation coverage. The factory body and seed semantics are unchanged. Additional merged scope: #3823 annotates AttackResultMetadata.from_metadata with Self and adds subclass runtime/assert_type coverage. Its factory body, defaults, unknown-role fallback, invalid-position handling and parent-linkage separation are unchanged. Additional merged scope: #3884 annotates TokenUsage.from_metadata with Self | None and checks subclass construction and static optional narrowing. Its factory body, absent-metadata None, counter filtering, extra counters and metadata round trips are unchanged. Additional merged scope: #3897 annotates RequestTraceContext.from_metadata with Self | None and checks subclass construction, optional narrowing, absent metadata and input non-mutation. Its runtime factory body, trace validation and metadata semantics are unchanged. Further merged contribution: #3992 completes subclass-preserving Self | None typing for ToolExecutionMetadata.from_metadata; runtime factory behavior is unchanged, with focused subclass/type and non-mutation coverage. Additional merged scope: #3999 annotates SeedIdentifier.from_seed with Self and adds base/subclass runtime and assert_type coverage. The final merged factory still delegates to cls.of(...); its executable body, behavioral parameter projection, hashes, promoted fields and serialization are unchanged. Slai.life (@huong8373tt-beep)'s reservation is released. No further focused case is identified here; propose a concrete subclass-preserving factory or fluent-API gap before claiming more work. The broad opportunity remains unchecked pending maintainer acceptance. Preserve fixed-base TagQuery.__and__/__or__ and ComponentIdentifier.with_eval_hash, independent generics and factories such as SeedPrompt.from_value_with_required_parameters that can return an existing base input.
  • P311-02: Improve dictionary contracts with TypedDict, Required, and NotRequired. Investigate internal metadata/provider payloads currently typed as dict[str, Any]. Distinguish absent keys from nullable values. RawAnalyticsProfile already provides prior art. Do not replace Pydantic runtime validation.
  • P311-03: Add exhaustive dispatch checks with assert_never / Never. Investigate closed enum/union dispatch for outcomes, states, modalities, or capabilities so added cases surface during type checking. Apply only where handling is genuinely exhaustive, not to intentionally open extension points. Merged scope: #3441 makes AttackStrategy._log_attack_outcome exhaustive across AttackOutcome, preserving existing log messages and rejecting unknown values. Additional merged scope: #3456 explicitly handles ScenarioResultView.CONVERSATIONS and guards both CLI dispatchers with assert_never, preserving HTML full-report override, default view, limits and lazy imports. Additional merged scope: #3472 makes AttackResultAnalytics._option exhaustive for all four AttackAnalyticsValueKind members, preserving display labels and stored keys, with an actual-source enum-extension type regression. Additional merged scope: #3504 explicitly handles all three PuzzleType members in PuzzledConverter.convert_async and adds an assert_never guard with a runtime regression. Constructor validation, crossword fallback, RNG behavior, Unicode handling, clues and output are preserved. Additional merged scope: #3601 covers all five CodeAttackConverter._encode encodings with a type-only assert_never guard, preserving the runtime invalid-state ValueError, constructor validation, template pairing and output. Per-encoding, invalid-state and actual-source type checks cover the change. Additional merged scope: #3654 adds an exhaustive assert_never guard for the validated RepeatTokenConverter mode dispatch, preserving its string-Literal API, None defaulting to split, validation errors/messages, punctuation, spacing, output and identifiers. The unreachable fallback uses the existing coverage exclusion pattern; changed-line coverage passed at 100%. Additional merged scope: #3797 makes PromptShieldTarget._input_parser exhaustive for its three declared Literal values with a TYPE_CHECKING-only assert_never. Shared parsing preserves forced-field early returns, None and unknown-runtime-field fallback, and trailing document text; a regression covers the fallback. Additional merged scope: #3830 makes HumanLabeledDataset.from_csv entry construction exhaustive for MetricsType.HARM and MetricsType.OBJECTIVE, with a type-only assert_never. CSV parsing, validation and the unknown-runtime-value objective fallback are preserved. Additional merged scope: #3924 makes ScorerEvaluator.from_scorer constructor selection exhaustive for both MetricsType members with a type-only guard. Omitted-type inference, explicit overrides and the historical unknown-runtime-value harm fallback are preserved. The final merged tests include the actual factory source checked against current and extended enums, addressing the coverage gap. Additional merged scope: #3998 makes TargetConfiguration.ensure_can_handle exhaustive for UnsupportedCapabilityBehavior RAISE/ADAPT with a type-only guard. Native-support and configured-normalizer early returns, missing-policy handling, error messages and the historical runtime fallback are preserved. The final tests check the actual method source against current and extended enums for Python 3.11/3.12, so removing the guard is detected. B Sainath Reddy (@Sainath-Reddy7)'s reservation is released. No further focused case is identified here; propose a concrete closed-dispatch gap before claiming more work. The broad opportunity remains unchecked pending maintainer acceptance. Keep open string, custom-callable and provider-extension dispatch unchanged.
  • P311-04: Add type-regression checks with assert_type. Investigate registry factories, generic results, decorators, and subclass-returning APIs. Ensure checks run through the type checker; assert_type is not a runtime assertion. Additional merged scope: #3625 adds assert_type(results, list[int]) for gather_with_cleanup_async and checks ordered values in tests/unit/test_async_utils.py. The canonical make ty checks tests/unit; the generic helper, public signature and cancellation/cleanup behavior are unchanged. Additional merged scope: #3686 adds concrete and protocol-consumer assert_type checks for DefaultInstanceRegistry[T].get, get_entry and get_all_instances, retaining T | None, RegistryEntry[T] | None and list[RegistryEntry[T]]. Present/missing retrieval, identity and sorted order are checked; the protocol consumer uses a helper parameter to avoid concrete assignment narrowing. Production signatures and runtime behavior are unchanged. Additional merged scope: #3770 adds four assert_type checks for the raw singleton-factory results of SlowConstructingRegistry and OtherSlowConstructingRegistry before runtime narrowing. Identity, subclass isolation, synchronization, timeouts and cleanup are preserved; production signatures and behavior are unchanged. Assessment: sufficiently addressed for this selective opportunity. Existing CI-checked coverage now protects generic gather results, concrete and protocol instance-registry retrieval, and subclass-returning factories including this singleton case. Controlled annotation regressions fail these checks. The bounded assessment found no additional worthwhile typing case to prescribe; this is not an exhaustive audit or a requirement to add assertions to every registry/generic API. The separate backend timing-test change in TEST: Assert Registry.get_registry_singleton return types (P311-04) #3770 is CI-reliability work, not P311-04 scope. Its traversal-oracle caveat does not reopen the completed typing case.
  • P311-05: Investigate LiteralString at static-template boundaries. Consider developer-authored SQL or command templates where the type checker can enforce the distinction. Confirm checker support. This is not runtime sanitization and never replaces parameterized SQL.
  • P311-06: Investigate TypeVarTuple / Unpack for variadic types. Look for a concrete heterogeneous-tuple or numerical/multimodal API that benefits. Do not add variadic complexity without a real need or assume this automatically checks NumPy shapes. Features standardized after 3.11 still need compatible backports.
  • P311-07: Investigate generic TypedDict / NamedTuple containers. Identify payload containers whose content type should be parameterized. Generic TypedDict in analytics result analysis provides existing prior art.

Async execution and diagnostics

  • P311-08: Enrich propagated exceptions with BaseException.add_note. Add useful execution, technique, dataset, or conversation context while preserving exception types and tracebacks. Notes appear in formatted tracebacks, not str(error). Exclude secrets and sensitive prompt content. Merged scope: #3175 adds dataset/provider notes to concurrent provider-fetch failures. Additional merged scope: #3277 adds scenario and atomic-attack names to caught worker-pool exceptions in Scenario._execute_atomic_attacks_parallel_async, preserving the original exceptions and existing single/grouped failure behavior. Regression coverage checks grouped notes and single-failure identity. Additional merged scope: #3281 labels cached-backend and analytics close failures in close_services_async, preserving the original exceptions, cancellation and single/grouped behavior. Additional merged scope: #3333 adds safe static phase notes for scenario-service shutdown and memory disposal in RuntimeLifecycle._shutdown_runtime_async. Existing notes, original exception identity/type, cleanup order, cancellation and single/grouped behavior remain intact. Additional merged scope: #3351 adds safe static session-release, active-client-stop and failed-startup-client-stop notes in GitHubCopilotTarget._cleanup_owned_resources_async, with regression coverage for phase labels and original errors. Complete: the identified dataset/provider, scenario-worker, backend service-close, runtime-shutdown and Copilot-cleanup context gaps are addressed. Current-code assessment found no further concrete, worthwhile propagation-context gap requiring another contribution for this item. Other examined groups already describe their operation or preserve ordered/chained causes; this is not a requirement to annotate every exception. Retain original exception identity/type/tracebacks, prior notes, cancellation, cleanup/retry ownership and single/grouped behavior; never add secrets or sensitive prompt content.
  • P311-09: Use scoped operation timeouts with asyncio.timeout. Investigate multi-await operations that should share one timeout scope instead of restarting a timeout for every step. Existing analytics, objective-generation, observation, and A2A usage provides prior art. Preserve cancellation and cleanup behavior. Merged scope: #3205 gives each Copilot receive loop one overall budget across streamed messages, with separate handshake/response budgets and cancellation/cleanup coverage. Completed: #3239 gives WebSocketTarget._initialize_connection_async one separate receive budget across all discarded initial messages, with aggregate-budget and cancellation/connection-cleanup coverage. Initialization sends, restoration, and normal responses retain their separate scopes. Current timeout-site assessment found no further worthwhile shared-budget migration: remaining wait_for calls bound individual operations, explicitly per-attempt probes, or already use a shared remaining deadline. Do not mechanically replace those calls.
  • P311-10: Propagate absolute deadlines with asyncio.timeout_at. Investigate nested operations/retries that should share one monotonic-clock deadline. Distinguish operation deadlines from per-step timeouts and keep cleanup semantics explicit. Merged scope: #3610 bounds _run_invocation_async in pyrit/_code_execution/docker_client.py with asyncio.timeout_at(deadline) during shielded Docker client startup and passes that same monotonic deadline into invocation handling. Capture and attached-client waiting already deducted elapsed time before this change; the new bound is startup. Startup-specific errors and cleanup ownership remain intact. Assessment: sufficiently addressed; no further worthwhile production migration was demonstrated in the inspected current timeout/retry surfaces. Existing A2A polling, WebSocket initialization, Copilot receive streaming and objective-generation execution already use shared scoped budgets, while QueryControl already computes one absolute deadline. Preserve independently bounded control commands and cancellation/container/workspace cleanup outside the expired invocation budget. The added test checks deadline wiring; behavioral expiry/cancellation regression coverage is an optional test follow-up, not an additional production migration or active reservation.
  • P311-11: Investigate structured concurrency with asyncio.TaskGroup. Identify parent-scoped child tasks with appropriate fail-fast semantics. The message-send service already uses it. It is not a blanket gather replacement or a concurrency limiter: sibling cancellation and exception shapes differ. Preserve ordered results and the deliberate cancellation/cleanup behavior in pyrit/common/task_utils.py. Merged scope: #3959 covers _AnthropicModelWrittenEvalsDataset._fetch_dataset_async: ordered file fetches now use an awaited owned task containing a TaskGroup. First-error identity and caller cancellation are preserved, including the Python 3.11/3.12 internal-cancellation-count regression; failure and outer cancellation drain async siblings. File/cache writes already running in threads cannot be interrupted. Further merged scope: #3980 covers Converter._convert_token_regions_async using the existing gather_with_cleanup_async helper, not a new TaskGroup migration. Failed or externally cancelled selected-region conversions drain async siblings while preserving result order, original errors and caller cancellation counts. File writes already running in threads remain non-interruptible. Víctor Fernández Ayllón (@vayllon301)'s reservation is released. No further focused case is identified here; propose a concrete remaining gap before claiming more work. The broad structured-concurrency opportunity remains unchecked pending maintainer acceptance.
  • P311-12: Investigate ExceptionGroup / except* handling. Preserve multiple failures and, where useful, handle matching subsets without losing other errors. Groups already exist in scenario execution and service shutdown. Carefully assess public exception contracts and cancellation (BaseExceptionGroup). Focused contribution merged: #3983 preserves shutdown errors and disposal for mixed groups, propagates live-apply cancellation groups, retains secondary Anthropic fetch failures as notes, and includes nested child failures and explicit causes in scenario diagnostics. Ordinary-error output and the original Anthropic exception contract remain unchanged, with focused regression coverage. Muhammad Waqar Anwar (@WaqarAnwar)'s reservation is released. No further focused case is identified here; propose a concrete remaining gap before claiming more work. The broad opportunity remains unchecked pending maintainer acceptance. Claiming the investigation does not preapprove behavior changes or blanket migrations.
  • P311-13: Investigate explicit task context with asyncio.create_task(..., context=...). Identify background operations that need selected tracing/execution ContextVars or a clean context. Default task creation already copies the current context; do not change it gratuitously. Investigation reported: @WaqarAnwar's findings report no current production bug and an execution-context carryover risk when a scheduled run fails in its own task rather than an attack worker. The report used a mock scenario and did not inspect every task creation call; this is not an independently verified production reproducer. A clean context for each scheduled run is proposed, not accepted. No change or completion is recorded; maintainer acceptance of the investigation is pending.
  • P311-14: Investigate asyncio.Runner for synchronous entry points. Consider multiple top-level calls sharing one managed loop/context in a shell or lifecycle harness. This does not permit nested loops inside an already-running notebook event loop. Preserve resource and loop ownership. Current investigation (claimed by Slai.life (@huong8373tt-beep)): Report finds no justified Runner change in the one-shot scan/migration CLIs, persistent shell loop, synchronous embedding facade or platform-specific browser-authentication loop. This is the contributor's reported conclusion, not independently verified acceptance. The opportunity remains unchecked pending maintainer acceptance; no mechanical substitution is requested.

Standard-library simplifications

  • P311-15: Parse project TOML with tomllib. Investigate replacing the version-extraction regex in build_scripts/stamp_compatibility.py with structured access to project.version. Preserve validation/provenance behavior. tomllib reads TOML; it does not write it. Completed: #3208 replaces the version regex with structured project.version parsing and covers table ordering, quoting, invalid metadata, and provenance errors.
  • P311-16: Simplify ISO timestamp parsing. Investigate removing trailing-Z compatibility rewrites, such as _RemoteDatasetLoader._parse_datetime, because 3.11 parses valid Z timestamps directly. Preserve timezone and accepted-input contracts. Do not edit immutable historical Alembic revisions merely for cleanup. Completed: #3176 removes the active loader shim and preserves invalid/non-string fallback behavior. Post-merge search found no other actionable ISO-parsing compatibility shims in pyrit or build_scripts; the remaining rewrite in historical migration d7e9f1a3b5c6 is intentionally immutable.
  • P311-17: Assess enum.StrEnum for appropriate enums. It is already used for harm categories and message-send models. Existing str, Enum migrations can change str(member) and formatting, and Ruff's automatic replacement is deliberately disabled. Focus on cases where string behavior is intended and compatibility is understood. Merged scope: #3850 migrates ConversationRetryReason to StrEnum. str() and formatting intentionally now return json_parsing instead of the class-qualified name; members, equality/hash/repr, JSON, value lookup and the ConversationRetry model remain compatible. Current focused case (claimed by Julio César Suástegui (@juliosuas)): investigate SeedOrigin in pyrit/models/seeds/seed_origin.py. Check qualified string/format consumers, equality/hash/repr, schema/JSON and persisted-seed round trips before deciding whether to migrate. Do not assume every existing string enum is safe.
  • P311-18: Investigate atomic regex groups and possessive quantifiers. Consider (?>...), *+, or ++ in a concrete scorer/parser pattern with unnecessary backtracking on large model outputs. Measure behavior and performance; these constructs change matching semantics and are not a universal regex-safety fix. Merged scope: #3702 uses atomic groups and whole-flag-token guards in ShellCommandOutputScorer's destructive rm branch. Historical detection cases and malformed/repeated-input regressions preserve the tested heuristic behavior; measured long-input families scale approximately linearly. Assessment: sufficiently addressed for this concrete opportunity. The actual squash and assessed current code have identical scorer/test files. This is not a proof for every regex/input or a requirement to migrate every pattern; retain command-separator behavior, /tmp exclusion and whole flag tokens. The scorer evaluates text, not shell execution. Additional merged scope: #3764 adds multiline Markdown-image detection with possessive scanner patterns and a CommonMark block parser. Reference normalization, code/HTML context, custom-pattern opt-out and bounded malformed-input regressions are covered. The actual merged scorer/tests and dependency files are unchanged in assessed current main. Existing regex matching is retained, so this is not complete CommonMark equivalence or a globally linear scorer claim. It does not change the completed shell assessment or require a blanket migration. The earlier unclaimed item overlap remains in the historical row; no reservation was retroactively granted. Additional merged scope: #3687 deliberately uses possessive shell-word/option scans and atomic operand alternatives while changing command-substitution detection semantics. Twelve checked-in malformed-input shapes time regex search below 0.5 seconds and separately check the public false verdict. This is supplemental performance-bounded use, not a semantics-preserving migration or a second completion. The inherited destructive-shell atomic pattern is unchanged and remains counted under FIX: Make ShellCommandOutputScorer destructive-shell regex linear time #3702. The merged shell scorer and dedicated tests match assessed current main; the general regex tests later changed default-pattern identity assertions separately. Shell-oracle and growth results in the PR are author-reported, not independently rerun here, and do not establish global linearity or atomic-versus-nonatomic equivalence. No blanket migration or new reservation is needed.
  • P311-19: Investigate hashlib.file_digest for whole-file hashes. Consider classifier artifact verification or other complete-file checks. Keep blocking I/O off async paths. Do not replace bounded evidence-reading loops that enforce budgets/cancellation and retain content. Merged scope: #3392 streams the two classifier compute_dataset_hashes() helpers through binary hashlib.file_digest handles, preserving SHA-256 values and dictionary order; multi-buffer regression tests cover current file contents. Assessment: sufficiently addressed. Keep _load_training_rows() hashing the same bytes it subsequently parses; keep LocalFileSurfaceSource._read_entry bounded/cancellable with retained evidence. Async serializer hashing supports storage backends that may be remote, memory media hashing reuses copied content, and FileDocumentStorage._compute_version receives bytes already in memory. No further worthwhile direct whole-file replacement identified; do not add synchronous local-file reads to async storage paths.
  • P311-20: Investigate inspect.getmembers_static for safe introspection. Identify registry inspection that should avoid invoking descriptors/properties. It intentionally bypasses dynamic lookup and lazy exports, so verify the discovery contract before adoption. Merged scope: #3531 statically inspects discover_in_directory modules, resolves only explicitly declared missing __all__ exports dynamically, and logs failed declared exports without losing other classes. Registration-outcome tests cover declared lazy exports with/without __dir__, unrelated dynamic names and failed exports; the earlier lazy-export regression is addressed. Additional merged scope: #3580 statically inspects InitializerRegistry._module_defined_initializers for single-file discovery and custom scripts, with targeted resolution of declared lazy exports. Own-module, base-class and abstract filters remain intact. Best-effort discovery logs failed declared exports; strict script loading and custom-script validation still reject them. Registration and strict/best-effort entry-point regressions cover these contracts. Assessment: the registry inspection opportunity is sufficiently addressed by fix: use static member discovery in registry inspection #3531 and FIX avoid dynamic attribute evaluation in single-file initializer discovery #3580; no further worthwhile blanket migration was found. Keep Registry._discover's intentional lazy-export materialization dynamic.

SQLite-specific memory opportunities

These must stay in the SQLite-specific memory/driver layer and preserve portable SQL Server behavior. Check what SQLAlchemy/aiosqlite exposes before choosing an implementation.

  • P311-21: Improve SQLite error handling using extended error codes/names. Analytics already checks sqlite_errorcode for plain SQLITE_BUSY. Investigate other real cases where structured codes are preferable to error-message matching, including distinctions such as stale snapshots. Completed scope: #3893 adds the driver's sqlite_errorcode/sqlite_errorname as deduplicated exception notes on propagated SQLite analytics failures, making distinctions such as SQLITE_BUSY_SNAPSHOT visible. Plain-BUSY retry policy, exception identity/messages and non-SQLite behavior are unchanged. The current bounded memory-handler survey found no worthwhile remaining message-based SQLite classification to replace; existing structured-code and domain-state handling stays intact. This completes the implemented diagnostics case, not a new retry policy or blanket database migration, and does not retroactively accept the earlier no-change proposal.
  • P311-22: Investigate SQLite connection getlimit / setlimit. Identify a real need for connection-level runtime limits and how limits interact with pooling and workloads. Avoid arbitrary limits that break existing behavior. Current focused case (claimed by Pedro Holanda (@AoiUltramar)): investigate SQLite runtime bind-variable limits and the conservative batching constants in MemoryInterface and MemorySession. The proposal and preliminary benchmark are not accepted implementation evidence. Verify limits on the actual pooled connection rather than assuming a separate probe connection represents it, preserve Azure SQL behavior, and report reproducible pinned-environment measurements before proposing a capped SQLite-only change or no-change conclusion.
  • P311-23: Investigate SQLite custom window functions. Assess whether an actual analytics use case benefits from create_window_function, with SQLite runtime support, driver integration, and portability considered. Prefer existing SQL when sufficient.
  • P311-24: Investigate incremental SQLite BLOB I/O. Assess whether any actual large-BLOB storage path benefits from blobopen. Do not redesign media storage just to use this API; a no-change conclusion may be appropriate.
  • P311-25: Investigate SQLite database serialization/deserialization. Assess a concrete snapshot/test/export use case for serialize / deserialize, including runtime availability, memory cost, concurrency, and driver boundaries.

Claim and PR tracking

The tracker is authoritative for per-item ownership; GitHub issue assignees are the aggregate set of active contributors. Available items have no active tracker row. Preserve released/expired rows as history when updating the tracker.

Item Contributor Claim comment / source Claimed on PR due PR Status
P311-14 Slai.life (@huong8373tt-beep) Claim comment 2026-11-06 2026-11-11 None yet Investigation reported no focused change; maintainer acceptance pending; GitHub assignment unavailable
P311-22 Pedro Holanda (@AoiUltramar) Claim comment 2026-11-06 2026-11-11 None yet Claimed; focused SQLite runtime-limit and batch-size investigation; GitHub assignment unavailable
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-11-06 2026-11-11 #3999 ✅ PR merged 2026-11-06; focused SeedIdentifier factory typing complete; reservation released
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-11-05 2026-11-10 #3992 ✅ PR merged 2026-11-06; ToolExecutionMetadata factory typing complete; reservation released; SeedIdentifier factory case available
P311-03 B Sainath Reddy (@Sainath-Reddy7) Claim comment 2026-11-05 2026-11-10 #3998 ✅ PR merged 2026-11-06; capability-behavior dispatch and typing regression complete; reservation released
P311-12 Muhammad Waqar Anwar (@WaqarAnwar) Claim comment 2026-11-05 2026-11-10 #3983 ✅ PR merged 2026-11-06; focused exception-group investigation complete; reservation released
P311-11 Víctor Fernández Ayllón (@vayllon301) Claim comment 2026-11-05 2026-11-10 #3980 ✅ PR merged 2026-11-06; selected-region sibling cleanup complete; reservation released
P311-11 Muhammad Waqar Anwar (@WaqarAnwar) Claim comment 2026-11-04 2026-11-09 #3959 ✅ PR merged 2026-11-05; focused Anthropic file-fetch case complete; reservation released; selected-region converter case available
P311-03 B Sainath Reddy (@Sainath-Reddy7) Claim comment 2026-11-04 2026-11-09 #3924 ✅ PR merged 2026-11-05; focused ScorerEvaluator dispatch and typing regression complete; reservation released; capability-behavior case available
P311-17 Julio César Suástegui (@juliosuas) Claim comment 2026-11-04 2026-11-09 None yet Claimed; focused SeedOrigin investigation reserved; GitHub assignment unavailable
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-11-03 2026-11-08 #3897 ✅ PR merged 2026-11-04; focused RequestTraceContext factory complete; reservation released; ToolExecutionMetadata case available
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-11-03 2026-11-08 #3884 ✅ PR merged 2026-11-03; focused TokenUsage factory complete; reservation released; RequestTraceContext case available
P311-17 Julio César Suástegui (@juliosuas) Claim comment 2026-11-02 2026-11-07 #3850 ✅ PR merged 2026-11-03; focused ConversationRetryReason case complete; reservation released; SeedOrigin investigation available
P311-03 Muhammad Waqar Anwar (@WaqarAnwar) Claim comment 2026-11-02 2026-11-07 #3830 ✅ PR merged 2026-11-03; dataset-entry dispatch complete; reservation released; evaluator dispatch available
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-11-02 2026-11-07 #3823 ✅ PR merged 2026-11-03; metadata factory typing complete; reservation released; TokenUsage case available
P311-02 XY (@XY-code1) Claim comment 2026-11-02 2026-11-07 None yet Claimed; focused internal payload investigation reserved; GitHub assignment unavailable
P311-21 B Sainath Reddy (@Sainath-Reddy7) Claim comment 2026-11-01 2026-11-06 #3893 ✅ PR merged 2026-11-04; SQLite codes/names diagnostics assessed complete; reservation released; earlier survey retained as background
P311-18 Jaafer Rahmani (@bIackr0se) PR opened without prior claim 2026-10-31 N/A; not assigned #3764 ✅ PR merged 2026-11-02; Markdown-image scope complete; historical unclaimed overlap with #3702 retained; no reservation granted
P311-18 Muhammad Waqar Anwar (@WaqarAnwar) PR opened without prior claim 2026-10-31 N/A; PR already open #3702 ✅ PR merged 2026-11-01; destructive-shell scope complete; reservation released; prior GitHub assignment unavailable
P311-03 Slai.life (@huong8373tt-beep) Claim comment 2026-10-31 2026-11-05 #3797 ✅ PR merged 2026-11-02; PromptShield dispatch complete; reservation released; dataset-entry dispatch available
P311-03 Slai.life (@huong8373tt-beep) Claim comment 2026-10-29 2026-11-03 #3654 ✅ PR merged 2026-10-31; RepeatToken dispatch complete; reservation released; PromptShield parser available
P311-04 Muhammad Waqar Anwar (@WaqarAnwar) Request comment 2026-11-01 N/A; not assigned #3770 ✅ PR merged 2026-11-01; singleton checks complete; no active reservation; opened while #3702 active (historical one-item conflict)
P311-04 mehmetl33t Claim comment 2026-10-30 2026-11-04 #3686 ✅ PR merged 2026-10-31; instance-registry checks complete; reservation released; singleton-factory checks available
P311-04 Slai.life (@huong8373tt-beep) Claim comment 2026-10-29 2026-11-03 #3625 ✅ PR merged 2026-10-29; gather result-type check complete; reservation released; instance-registry type checks available
P311-01 Julio César Suástegui (@juliosuas) Claim comment 2026-11-01 2026-11-06 #3779 ✅ PR merged 2026-11-02; adversarial-prompts factory complete; reservation released; attack-result metadata factory available
P311-01 Julio César Suástegui (@juliosuas) Request comment 2026-10-28 2026-11-02 #3626 ✅ PR merged 2026-10-31; WebSearchConfig factory complete; reservation released; adversarial-prompts factory available
P311-01 Slai.life (@huong8373tt-beep) Claim, withdrawal 2026-10-29 N/A None Withdrawn 2026-10-29 before assignment; no reservation
P311-10 Slai.life (@huong8373tt-beep) Claim comment 2026-10-28 2026-11-02 #3610 ✅ PR merged 2026-10-29; Docker sandbox invocation deadline complete; opportunity complete; reservation released
P311-20 Slai.life (@huong8373tt-beep) Claim comment 2026-10-27 2026-11-01 #3580 ✅ PR merged 2026-10-28; single-file inspection complete; opportunity complete; reservation released
P311-03 Julio César Suástegui (@juliosuas) Claim comment 2026-10-27 2026-11-01 #3601 ✅ PR merged 2026-10-28; CodeAttack encoding dispatch complete; reservation released; RepeatToken mode dispatch available
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-10-26 2026-10-31 #3577 ✅ PR merged 2026-10-27; JSON response factory complete; reservation released; web-search policy factory available
P311-20 Slai.life (@huong8373tt-beep) Claim comment 2026-10-25 2026-10-30 #3531 ✅ PR merged 2026-10-26; directory discovery complete; reservation released; single-file initializer inspection available
P311-01 Julio César Suástegui (@juliosuas) Claim comment 2026-10-25 2026-10-30 #3529 ✅ PR merged 2026-10-26; HarmDefinition factory complete; reservation released; JSON response factory available
P311-01 Niladri Das (@bniladridas) Claim comment 2026-10-24 2026-10-29 #3520 ✅ PR merged 2026-10-25; run-size estimate factory complete; HarmDefinition factory available, reservation released
P311-03 Slai.life (@huong8373tt-beep) Claim comment 2026-10-24 2026-10-29 #3504 Closed without merging 2026-10-25; reopened 2026-10-26; ✅ PR merged 2026-10-26; puzzle dispatch complete; reservation released; CodeAttack encoding dispatch available
P311-03 Slai.life (@huong8373tt-beep) Claim comment 2026-10-23 2026-10-28 #3472 ✅ PR merged 2026-10-23; analytics option-kind dispatch complete; puzzle dispatch available, reservation released
P311-01 Julio César Suástegui (@juliosuas) Claim comment 2026-10-22 2026-10-27 #3470 ✅ PR merged 2026-10-23; TagQuery factory family complete; run-size estimate factory available, reservation released
P311-03 Slai.life (@huong8373tt-beep) Claim comment 2026-10-22 2026-10-27 #3456 ✅ PR merged 2026-10-23; CLI result-view dispatch complete; analytics option dispatch available, reservation released
P311-03 Slai.life (@huong8373tt-beep) Claim comment 2026-10-21 2026-10-26 #3441 ✅ PR merged 2026-10-22; AttackOutcome logging complete; CLI view dispatch available, reservation released
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-10-21 2026-10-26 #3411 ✅ PR merged 2026-10-21; placeholder factory complete; further TagQuery family available, reservation released
P311-01 Slai.life (@huong8373tt-beep) Claim comment 2026-10-20 2026-10-25 #3383 ✅ PR merged 2026-10-21; factory family complete; remaining MessagePiece case available, reservation released
P311-19 ARJUN K (@arjun-k047) Claim comment 2026-10-20 2026-10-25 #3392 ✅ PR merged 2026-10-20; P311-19 complete; reservation released
P311-08 Slai.life (@huong8373tt-beep) Claim comment 2026-10-19 2026-10-24 #3351 ✅ PR merged 2026-10-20; opportunity complete; reservation ended
P311-08 Slai.life (@huong8373tt-beep) Claim comment 2026-10-19 2026-10-24 #3333 ✅ PR merged 2026-10-19; contribution complete; Copilot cleanup-phase investigation available, reservation released
P311-08 Slai.life (@huong8373tt-beep) Claim comment 2026-10-17 2026-10-22 #3281 ✅ PR merged 2026-10-19; contribution complete; runtime-shutdown phase context remains available, reservation released
P311-08 Slai.life (@huong8373tt-beep) Claim comment 2026-10-16 2026-10-21 #3277 ✅ PR merged 2026-10-17; contribution complete; service-shutdown context remains available, reservation released
P311-01 成功したナイロンカエル (@vito233) Claim comment 2026-10-16 2026-10-21 #3266 ✅ PR merged 2026-10-17; contribution complete; further model factory cases available, reservation released
P311-09 Aizen__28 (@Yudeeswaran) Claim comment 2026-10-15 2026-10-20 #3239 ✅ PR merged 2026-10-16; opportunity complete; reservation ended
P311-15 Shivam Maurya (@shivampwm2020) Claim comment 2026-10-14 2026-10-19 #3208 ✅ PR merged 2026-10-14; opportunity complete; reservation ended
P311-01 Kushal Talati (@kushaltalati) Claim comment 2026-10-14 2026-10-19 #3201 ✅ PR merged 2026-10-14; contribution complete; further model factory cases available, reservation released
P311-09 Aizen__28 (@Yudeeswaran) Claim comment 2026-10-14 2026-10-19 #3205 ✅ PR merged 2026-10-14; contribution complete; initialization timeout investigation available, reservation released
P311-01 Kushal Talati (@kushaltalati) Claim comment 2026-10-13 2026-10-18 #3194 ✅ PR merged 2026-10-14; contribution complete; remaining cases available, reservation released
P311-08 Aizen__28 (@Yudeeswaran) Claim comment 2026-10-13 2026-10-18 #3175 ✅ PR merged 2026-10-13; contribution complete; remaining cases available, reservation released
P311-16 Kushal Talati (@kushaltalati) Claim comment 2026-10-13 2026-10-18 #3176 ✅ PR merged 2026-10-13; opportunity complete; reservation ended
P311-04 Roman Lutz (@romanlutz) Historical PR, opened before this issue; no claim N/A N/A #3064 ✅ PR merged; prior art covers part of the opportunity; remaining cases available

Describe alternatives you've considered, if relevant

Keep existing implementations when they already express the required behavior clearly. Avoid a repo-wide modernization sweep: each contributor investigates and opens a focused PR for ONE opportunity at a time. Separate changes within an item should remain cohesive; coordinate here before splitting or expanding scope.

Additional context

  • Python 3.11 release notes
  • Article motivating this inventory
  • Existing adoption includes datetime.UTC, Self, NotRequired, generic TypedDict, StrEnum, TaskGroup, timeouts, exception groups, and SQLite extended error codes.
  • Better traceback locations and interpreter improvements arrive automatically on 3.11+; they do not need modernization PRs, and model-call performance is often network-bound.
  • typing.override, typing.TypeAliasType, the type Alias = ... statement, and class Foo[T] syntax are 3.12+, not 3.11. Keep needed typing_extensions imports while supporting 3.11.
  • Other 3.11 APIs such as dataclass_transform can be considered when there is a concrete fitting abstraction, but should not motivate new abstractions on their own.

Activity

  1. romanlutz commented on Oct 13, 2026

    @romanlutz
    ContributorAuthor

    Roman Lutz (@romanlutz): P311-04 has related work in #3064, which opened on 2026-10-09 at 22:00:12 UTC, before this tracking issue existed, and merged on 2026-10-13 at 07:06:53 UTC. There was no prior claim here, so this is historical overlap, not a contributor bypassing the new claim rule.

    That PR adds assert_type checks for retry helpers and async task return types, and brings the unit tests into the make ty check. It partially addresses the type-regression opportunity. I am linking it as prior art but leaving P311-04 unchecked and available pending your assessment of the remaining scope. No duplicate claim exists and no new assignment is needed for the already-merged PR.

  2. Yudeeswaran commented on Oct 13, 2026

    @Yudeeswaran
    Contributor

    Claiming P311-08 (BaseException.add_note).

    I’ll investigate dataset-fetch failure propagation in SeedDatasetProvider.fetch_datasets_async (pyrit/datasets/seed_datasets/seed_dataset_provider.py). Providers are fetched concurrently, and a failing provider’s exception may propagate without identifying the dataset (e.g., “SeedDataset cannot be empty”).

    I plan to add diagnostic context using add_note(), including the dataset name and provider class, while preserving the original exception type and traceback.

    I’ll add a regression test to verify that the original exception is preserved and that the diagnostic note contains the expected dataset and provider information, without exposing prompt content or other sensitive data.

  3. romanlutz commented on Oct 13, 2026

    @romanlutz
    ContributorAuthor

    Assigning P311-08 to you Aizen__28 (@Yudeeswaran).

  4. kushaltalati commented on Oct 13, 2026

    @kushaltalati
    Contributor

    Claiming P311-16 (Simplify ISO timestamp parsing).
    I'll remove the .replace("Z", "+00:00") workaround in _parse_datetime since Python 3.11 handles it natively. Historical Alembic revisions will remain untouched. I'll preserve timezone and accepted-input contracts and update test coverage.

  5. romanlutz commented on Oct 13, 2026

    @romanlutz
    ContributorAuthor

    Assigning P311-16 to you Kushal Talati (@kushaltalati). Linked #3176 and Aizen__28 (@Yudeeswaran)'s #3175 in the table.

  6. romanlutz commented on Oct 13, 2026

    @romanlutz
    ContributorAuthor

    Post-merge assessment for P311-08: #3175 is merged and successfully covers dataset/provider context on concurrent dataset-fetch failures. Thank you Aizen__28 (@Yudeeswaran); your contribution is complete and your reservation is released, so you may claim another opportunity.

    The broader exception-note opportunity remains unchecked and available. I assessed the merged diff and current main at 20b35221c514b7f0898caac12f14b71631889027 and added two concrete follow-up cases to the checklist:

    • Scenario._run_atomic_attacks_parallel_async: caught errors are stored without their atomic-attack association before aggregation. Safe scenario/atomic-group notes could identify which concurrent operation failed without logging seed text.
    • close_services_async: exception notes could identify which close operation produced each grouped shutdown error.

    These are focused investigations, not instructions to annotate every exception. Preserve cancellation, exception identities/types, group contracts, and sensitive-data boundaries. Claim exactly one opportunity on this thread before starting.

  7. kushaltalati commented on Oct 13, 2026

    @kushaltalati
    Contributor

    PR #3176 is approved with all CI checks passing. Once merged, I would like to claim P311-01 (Extend typing.Self adoption).

    I plan to investigate adopting typing.Self across classmethod factories and fluent methods (such as YamlLoadable.from_yaml_file and Seed.from_yaml_file) while preserving independent generic parameters.

  8. romanlutz commented on Oct 13, 2026

    @romanlutz
    ContributorAuthor

    P311-16 is complete with #3176. No other active ISO-parsing workarounds remain; the historical migration stays unchanged. Assigning P311-01 to you Kushal Talati (@kushaltalati) now that #3176 is merged.

  9. 203 remaining items

  10. Sainath-Reddy7 commented on Nov 6, 2026

    @Sainath-Reddy7
    Contributor

    PR for the reserved P311-03 TargetConfiguration.ensure_can_handle case: #3998 (opened within the claim window, due 2026-11-10).

  11. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    #3992 completes the ToolExecutionMetadata case. Thanks Slai.life (@huong8373tt-beep); your reservation is released. P311-01 remains open for the SeedIdentifier factory case described above. Linked the timely P311-03 PR #3998, B Sainath Reddy (@Sainath-Reddy7).

  12. huong8373tt-beep commented on Nov 6, 2026

    @huong8373tt-beep
    Contributor

    Claiming P311-01, the focused SeedIdentifier.from_seed case described in the tracker.

    I will investigate changing the factory return annotation to Self, since it delegates to cls.of(...), which constructs through the subclass. I will add focused subclass runtime and assert_type coverage while preserving behavioral parameter projection, content/evaluation hashes, promoted fields, serialization, and the existing runtime factory behavior. I will keep the scope limited to this factory and its directly coupled tests, and will open the PR within the claim window.

    Slai Agent Runtime performed the investigation, implementation, and test execution. The account owner authorized this contribution and is responsible for its content.

  13. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    #3983 is merged. Thanks Muhammad Waqar Anwar (@WaqarAnwar); the focused exception-group investigation is complete and your reservation is released. P311-12 remains unchecked pending maintainer acceptance of the broader opportunity; any further claim should identify a concrete remaining gap.

  14. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    #3980 is merged. Thanks Víctor Fernández Ayllón (@vayllon301); selected-region sibling cleanup is complete and your reservation is released. It reuses the existing cleanup helper rather than adding TaskGroup. P311-11 remains unchecked pending maintainer acceptance of the broader opportunity.

  15. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    Assigning P311-01 to you Slai.life (@huong8373tt-beep) and linking #3999. Please change its Closes #3166 reference to Relates to #3166; this PR completes one focused case, not the whole checklist.

  16. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    #3998 is merged. Thanks B Sainath Reddy (@Sainath-Reddy7); the capability-behavior dispatch and its typing regression are complete, and your reservation is released. P311-03 remains unchecked pending maintainer acceptance of the broader opportunity.

  17. WaqarAnwar commented on Nov 6, 2026

    @WaqarAnwar
    Contributor

    Hello Roman Lutz (@romanlutz) ,

    P311-13: Investigation result

    Is there a bug? Not today, but there is one weak spot.

    What I checked: a background task gets a copy of its creator's context, so a queued scenario run could inherit values from the failed run before it. I traced all 18 ContextVars in pyrit/ and the main long-lived background tasks, but didn't read every create_task call. I also tested failed runs followed by a queued run, using a mock scenario that copies the real structure.

    What I found:

    • Most variables: 17 of the 18 are reset or cleared when their work ends, so they don't carry over.
    • Execution context: this one deliberately stays set after an exception. Scenario attacks run in worker tasks, so a leftover value stays in the worker. In the test, the next queued run started clean.
    • Tracing: target spans already start from an empty Context(), so nothing is needed here.

    The weak spot: when the test failed in the run's own task instead of a worker, the value did reach the next queued run. It could then appear in that run's error details. No scenario in the repo fails this way today, but one that ran attacks directly in its own task could.

    Recommendation: no change is needed for current behaviour. If that weak spot needs closing, starting each scheduled run with create_task(..., context=Context()) would do it.

  18. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    Thanks Muhammad Waqar Anwar (@WaqarAnwar). I've linked your investigation under P311-13. Its no-change conclusion remains pending maintainer acceptance, so the item stays unchecked.

  19. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    #3999 is merged. Thanks Slai.life (@huong8373tt-beep); the SeedIdentifier factory case is complete and your reservation is released. P311-01 remains unchecked pending maintainer acceptance of the broader opportunity.

  20. huong8373tt-beep commented on Nov 6, 2026

    @huong8373tt-beep
    Contributor

    Claiming P311-14 (asyncio.Runner for synchronous entry points).\n\nI will investigate PyRIT synchronous entry points and determine whether multiple top-level calls currently create/dispose separate event loops where one managed asyncio.Runner would preserve lifecycle/context more safely. I will keep notebook/already-running-loop behavior unchanged, preserve resource and loop ownership, and only open a focused PR if a concrete improvement is justified, with regression coverage.\n\nSlai Agent Runtime performed the investigation, implementation, and test execution. The account owner authorized this contribution and is responsible for its content.

  21. huong8373tt-beep commented on Nov 6, 2026

    @huong8373tt-beep
    Contributor

    P311-14 investigation result — no focused code change justified\n\nI inspected the current synchronous entry points and their ownership/lifecycle contracts:\n\n- pyrit_scan.main and pyrit_migrate.main each execute one top-level coroutine and then exit. Replacing their one-shot asyncio.run(...) with asyncio.Runner would not share a loop or context across calls and would not change resource ownership.\n- pyrit_shell.PyRITShell already owns one persistent event loop on its background thread and schedules commands onto it with run_coroutine_threadsafe; changing this to a Runner would not improve the existing lifecycle.\n- OpenAITextEmbedding.generate_text_embedding is a synchronous facade over an async client. A reusable Runner would require a new long-lived runner/client shutdown contract, while a per-call Runner has the same one-shot semantics. It also would not make this facade safe to call from an already-running event loop, which this opportunity explicitly does not target.\n- The browser-authenticator thread deliberately creates a platform-specific loop (Proactor on Windows) for Playwright and runs one operation; a generic Runner replacement would risk changing that loop policy without providing a shared-call benefit.\n\nConclusion: the current production surfaces do not have a concrete multi-call synchronous lifecycle gap where asyncio.Runner improves behavior without introducing a new ownership contract. I am not opening a code PR for a mechanical substitution.\n\nSlai Agent Runtime performed the investigation. The account owner authorized this contribution and is responsible for the report.

  22. AoiUltramar commented on Nov 6, 2026

    @AoiUltramar

    Claiming P311-22 (getlimit/setlimit).

    Concrete case: MemoryInterface._MAX_BIND_VARS = 500 (and the duplicate in MemorySession) is documented as a conservative default based on SQLite's 999 limit, which was the default before SQLite 3.32; current builds default to 32766 and some ship higher (Debian/Kali: 250000).

    Preliminary findings from local runs: with the pinned SQLAlchemy/aiosqlite, getlimit isn't exposed on the async adapter or aiosqlite's public connection; the clean option would be probing a short-lived sqlite3 connection. MemorySession is shared with Azure SQL and its batch size doesn't depend on dialect, so raising it would need a dialect check; I'd leave it at 500. Early benchmarks (unpinned environment) of get_message_pieces with 10k IDs show the SQL portion dropping by ~20–26 ms of ~700 ms per call (~3–4%); end-to-end timings were too noisy (±200 ms) to show a reliable difference.

    Plan: re-run the benchmarks in a uv sync --frozen environment with more repetitions, then report either a capped SQLite-only change or a no-change conclusion here.

  23. romanlutz commented on Nov 6, 2026

    @romanlutz
    ContributorAuthor

    Assigning P311-14 to you Slai.life (@huong8373tt-beep) and P311-22 to you Pedro Holanda (@AoiUltramar). GitHub assignment is unavailable, so both reservations are recorded in the table. Huong's no-change report is linked; P311-14 stays unchecked pending maintainer acceptance.

  24. AoiUltramar commented on Nov 6, 2026

    @AoiUltramar

    Update on P311-22: the data ended up pointing to a small, capped change rather than a no-change result.

    What I measured (uv sync --frozen, SQLite 3.46.1, one VM, interleaved runs, 40–80 samples per cell, get_message_pieces): bumping SQLite's _MAX_BIND_VARS from 500 to 2000 makes the async path ~11% faster at 10k IDs and ~17% faster at 2k IDs (the p25–p75 ranges don't overlap at 2k). Sync barely moves. Going above 2000 only helps calls with more than 2000 IDs (async at 10k IDs: 260 → 240 ms with a limit of 10000), and it would put SQLite well above Azure's 2000, so I'd stop there.

    What I'm planning: SQLiteMemory._MAX_BIND_VARS = 2000 when getlimit(SQLITE_LIMIT_VARIABLE_NUMBER) >= 2000, and 500 otherwise, so older builds behave exactly as today. Neither the async adapter nor aiosqlite exposes getlimit publicly, so I'd read it from a short-lived sqlite3 connection. MemorySession (500) and Azure's _MAX_BIND_VARS (2000) stay as they are, and test_batching_scale.py will size its inputs from the SQLite value so batching stays covered.

    One thing I'd like your call on: raising the limit broke _execute_get_prompt_scores. Its JSON OR batches cost 2 bind params per term, because the JSON path repeats in every term under the qmark drivers (I confirmed this by compiling the real SQLite and Azure builders). At 1999 terms, SQLite rejects the query with Expression tree is too large (maximum depth 1000), so the PR needs to bound json_batch_size.

    The same math says Azure is probably already affected: 1999 terms is ~3998 params, over SQL Server's 2100 limit, so get_prompt_scores with more than ~1050 IDs likely fails there today. I haven't been able to verify that against a live Azure instance.

    Bounding it as min(_MAX_BIND_VARS, 500) - 1 (499 terms: 998 binds, depth ~499) keeps SQLite's current behavior. Locally, it fixes the SQLite failure: the failing test passes at 2000, and the batching suite still passes at 500. On paper it should fix Azure too (998 params). Would you like that included in the P311-22 PR, or should I file the Azure part as its own issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

help wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions