Skip to content

gh-92041: Resolve registered frames from verified globals ownership - #160254

Open
mdeck wants to merge 5 commits into
python:mainfrom
mdeck:inspect-registered-frame-fastpath-20261022
Open

mdeck wants to merge 5 commits into
python:mainfrom
mdeck:inspect-registered-frame-fastpath-20261022

Conversation

@mdeck

@mdeck mdeck commented Oct 22, 2026 •

Copy link
Copy Markdown

Resolve registered frames and tracebacks directly from their globals owner when the registered module identity and normalized filename match. This avoids scanning sys.modules and gives a verified owner consistent precedence over a conflicting filename-cache entry.

Successful ownership resolution does not populate the shared filename map, so bare code-object lookup remains filename based and is not changed by this fast path. A cached exact module with the frame's identical globals can be returned immediately: it cannot conceal a different globals owner, and the existing filename fallback would return that same module. Other cached results still require ownership verification.

Private execution namespaces, mismatched origins, module subclasses, dynamic filenames, fileless modules and unrelated filename overrides retain the filename fallback. Source/bytecode normalization is preserved. Source retrieval is unchanged, including linecache.getlines() overrides and the module globals passed to them. Documentation and NEWS describe the ownership precedence directly.

This implements the frame/traceback split discussed in #92042. It remains independent of and complementary to #156685; the changes compose cleanly in either order. No registry watchers, C changes or negative caching are included.

Performance

Measurements use source-pinned variants in one CPython 3.16 -O3 Linux build without PGO/LTO. Eight fresh processes use counterbalanced ABBA/BAAB batches; intervals are 95% process-level bootstrap intervals, with all samples retained and identical-code controls.

  • Direct registered-frame lookup with warm filename caches: about 2.04 µs on the pinned mainline versus 0.40 µs with this revision; paired time ratio 0.199 (0.195–0.203), approximately 5× faster.
  • Four-frame source-context stack with warm filename/source caches: paired time ratio 1.002 (0.992–1.012) versus mainline. No statistically resolved regression; smaller effects remain unresolved.
  • The same stack with gh-92041: Avoid a second sys.modules scan in inspect.getmodule #156685 composed: ratio 1.006 (0.996–1.016) versus that replacement alone.
  • Explicit cached frame lookup: about 0.271 → 0.266 µs. Ordinary-function lookup costs about 11 ns more; warm code-object lookup about 15 ns more.
  • Ownership verification has costs on fallback paths: explicit cached private-namespace lookup about 0.275 → 0.508 µs, and a registered frame executing a different source file about 0.280 → 1.512 µs. These paths are not claimed as performance improvements.

These are microbenchmark results for the specified workloads, not an application-wide speedup. Earlier cold-registry measurements linked in the discussion concern earlier source snapshots and are not presented as measurements of this revision.

Validation

  • Six suites pass standalone and composed with gh-92041: Avoid a second sys.modules scan in inspect.getmodule #156685: inspect, zipimport support, import, linecache, traceback and pydoc. Each run reports 1,149 tests, 4 skipped, with --fail-env-changed.
  • 648 cold differential cases preserve mainline behavior outside the intended collision change. 5,184 cached cases preserve the preceding ownership revision's results.
  • Regression coverage includes conflicting cached modules, module subclasses spoofing globals, source/bytecode aliases, explicit filename overrides, registry replacement and linecache.getlines() hooks receiving the frame owner's globals.
  • Configured lint hooks and git diff --check pass.
  • Native build, normal documentation HTML and changelog builds pass. The project's documentation warning gate passes; existing baseline nitpicky warnings remain.

Developed and tested with Codex (AI assistance).

mdeck added 2 commits October 22, 2026 09:44
Resolve a frame or traceback from its registered globals owner only when
its canonical filename matches. Reuse the existing filename cache and
retain the fallback for other objects and namespaces.

Developed and tested with Codex (AI assistance).
The fixture left a stale __file__ value beneath a module subclass property.
LOAD_ATTR_MODULE specialization can read that value, making the assertion
depend on warmup state in the existing filename fallback, including main.

Remove the shadow value and exercise warmed getfile access, two changing
origins, and independent cold frame/traceback lookups. Keep production code
unchanged.

Verified the corrected test 50 times on each of main and the candidate;
all six focused suites pass (1,135 tests reported, four skipped).

mdeck commented Oct 22, 2026

Copy link
Copy Markdown
Author

@eendebakpt, would you be willing to review this follow-up to #92042? It implements the frame/traceback split you suggested, without the registry watchers or C changes, and is independent of and complementary to #156685.

CI is green; regression coverage and reproducible benchmarks are linked in the description. I’d particularly appreciate your view on verified frame globals taking precedence for an uncached filename collision. Thanks for your earlier feedback.

Comment thread Lib/inspect.py Outdated
Comment on lines +947 to +948
except (TypeError, FileNotFoundError):
pass

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
except (TypeError, FileNotFoundError):
pass
except FileNotFoundError, TypeError:
pass
return None

Comment thread Lib/inspect.py Outdated
Comment on lines +940 to +942
if (type(module) is not types.ModuleType or
module.__dict__ is not namespace or '__getattr__' in namespace or
type(namespace.get('__file__')) is not str):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (type(module) is not types.ModuleType or
module.__dict__ is not namespace or '__getattr__' in namespace or
type(namespace.get('__file__')) is not str):
if (type(module) is not types.ModuleType
or module.__dict__ is not namespace
or '__getattr__' in namespace
or type(namespace.get('__file__')) is not str):

Comment thread Lib/inspect.py Outdated
Comment on lines +968 to +969
frame = object if isframe(object) else (
object.tb_frame if istraceback(object) else None)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
frame = object if isframe(object) else (
object.tb_frame if istraceback(object) else None)
if isframe(object):
frame = object
elif istraceback(object):
frame = object.tb_frame
else:
frame = None

@picnixz
picnixz requested a review from eendebakpt October 22, 2026 18:23
Comment thread Lib/inspect.py Outdated
module = sys.modules.get(name)
# Leave dynamic module metadata to the existing filename fallback.
if (type(module) is not types.ModuleType or
module.__dict__ is not namespace or '__getattr__' in namespace or

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is '__getattr__' in namespace ever needed?

@property
def __file__(self):
return self.source
# Do not leave a shadow value that LOAD_ATTR_MODULE can read instead

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be #156474

@@ -0,0 +1,3 @@
Speed up :func:`inspect.getmodule` for frames and tracebacks whose globals
belong to a registered module with a matching filename. The globals owner

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This promises a bit more than the code does. For example:

import inspect, os, sys, tempfile, types

fn = os.path.join(tempfile.mkdtemp(), 'owner.py')
open(fn, 'w').close()
owner = types.ModuleType('owner_mod'); owner.__file__ = fn
other = types.ModuleType('other_mod'); other.__file__ = fn
sys.modules['owner_mod'] = owner
sys.modules['other_mod'] = other        # inserted later -> wins a registry scan

exec(compile("import sys\nframe = sys._getframe()\n", fn, 'exec'), owner.__dict__)
frame = owner.__dict__['frame']

print("1 frame, cold          :", inspect.getmodule(frame).__name__)
inspect.getmodule(compile("pass", "<nowhere>", "exec"))   # unrelated miss
print("2 frame, after a miss  :", inspect.getmodule(frame).__name__)
inspect.modulesbyfile.clear(); inspect._filesbymodname.clear()
print("3 frame, cold again    :", inspect.getmodule(frame).__name__)
inspect.modulesbyfile.clear(); inspect._filesbymodname.clear()
print("4 code object, cold    :", inspect.getmodule(frame.f_code).__name__)
inspect.modulesbyfile.clear(); inspect._filesbymodname.clear()
inspect.getmodule(frame)
print("5 code obj after frame :", inspect.getmodule(frame.f_code).__name__)

This is a pathological case (intentionally giving two modules the same filename), so maybe fine to keep the code as is. But rewrite the news entry.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please indicate this change in the docs of inspect directly. This may be breaking for some people.

@read-the-docs-community

read-the-docs-community Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

…hits

Preflight the ownership-only revision before updating python#160254.
Keep source retrieval unchanged and preserve linecache.getlines overrides.

Developed and tested with Codex (AI assistance).
@mdeck mdeck changed the title gh-92041: Avoid registry scans for uncached registered frames gh-92041: Resolve registered frames from verified globals ownership Oct 24, 2026

mdeck commented Oct 24, 2026 •

Copy link
Copy Markdown
Author

Updated in 6a91668 to give verified frame/traceback globals ownership consistent precedence over conflicting filename-cache entries. Successful ownership matches no longer populate the shared filename map; bare code-object lookup remains filename based. The documentation and NEWS now state this behavior directly.

A narrow cached-module/globals identity check retains the cheap warm path. Source retrieval is unchanged, including linecache.getlines overrides and their module-globals argument. The revision remains independent of #156685 and composes cleanly in either order.

All six relevant suites pass standalone and composed (1,149 reported, 4 skipped each); configured lint and the documentation warning gate pass. Updated source-pinned measurements and fallback costs are in the description: direct registered-frame lookup is approximately 5× faster than the pinned mainline, while the warm four-frame source-context stack has no statistically resolved regression (time ratio 1.002, 95% interval 0.992–1.012).

mdeck commented Oct 25, 2026

Copy link
Copy Markdown
Author

I measured the current revision’s cold-cache behavior, including with/without #156685. The main benefit is avoiding registry scans on first inspection and when encountering a newly registered module.

Four-frame inspect.stack(context=1), milliseconds per call:

Cache state / registry entries Mainline #156685 This PR Both PRs
Filename and source caches empty / 125 1.567 1.530 0.140 0.106
Filename and source caches empty / 1,125 9.303 9.556 0.125 0.138
Filename and source caches empty / 10,125 86.152 86.007 0.141 0.137
New registered module after filename-cache warm-up; source warm / 1,125 0.303 0.309 0.073 0.060
Same new-module case / 10,125 2.118 2.112 0.074 0.077
Warm filename and source caches / 1,125 0.046 0.046 0.046 0.046
Empty caches, one private-namespace frame / 1,125 11.803 11.736 11.962 11.840

At 1,125 entries, this saves about 9 ms on first inspection and 0.23 ms when encountering the new module. These are savings on particular inspection calls, not a sustained application-wide speedup. One private-namespace frame still triggers filename fallback and largely removes the cold-cache benefit. #156685 does not eliminate that initial scan in this workload.

Measurements use pinned inspect.py sources from mainline e60a9da9919b and this PR 6a916683df63, each with/without #156685’s implementation at 0a125b297579, loaded into the same CPython 3.16 -O3 Linux build without PGO/LTO. Eight fresh worker processes per registry size, randomized mirrored variant order, and eight single-call cold samples per variant/process. Cache reset, module setup and thread startup are outside timing.

“Cold” means empty inspect filename dictionaries and linecache; filesystem caches remain warm. The initial 125-entry registry includes benchmark setup. Larger cases add synthetic registered modules pointing to existing source files; 10,125 entries is a scaling stress test. The new-module case excludes import execution.

A longer warm-only run shows no resolved difference: paired time ratio 0.996 (95% interval 0.974–1.021) versus mainline and 1.008 (0.994–1.025) versus #156685 alone. Identical-source controls are consistent with no change. Source-context checks agree across variants, and separate instrumentation confirms that verified registered-frame lookup avoids the registry scan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants