Repository navigation
asyncio: ProactorEventLoop busy-loops at 100% CPU forever when the self-pipe socketpair reaches EOF #156333
Description
Bug report
Bug description:
On Windows, BaseProactorEventLoop wakes itself through a self-pipe, which is a loopback TCP socketpair created by socket.socketpair(). If that connection reaches a clean EOF while the loop is running -- for example the OS tears the idle loopback connection down across a power/session state change -- the loop spins at 100% of one core forever, with no exception raised and nothing logged.
Lib/asyncio/proactor_events.py:
def _loop_self_reading(self, f=None):
try:
if f is not None:
f.result() # may raise
if self._self_reading_future is not f:
return
f = self._proactor.recv(self._ssock, 4096)
except exceptions.CancelledError:
return
except (SystemExit, KeyboardInterrupt):
raise
except BaseException as exc:
self.call_exception_handler({
'message': 'Error on reading from the event loop self pipe',
'exception': exc,
'loop': self,
})
else:
self._self_reading_future = f
f.add_done_callback(self._loop_self_reading)At EOF, f.result() returns b''. That is not an exception, so no except branch runs and control falls through to else, which re-arms recv() on a socket that is at EOF. That recv completes immediately, whose callback is _loop_self_reading, which re-arms again -- a tight infinite loop.
The loop is otherwise completely idle. py-spy shows MainThread as active+gil with:
_loop_self_reading (asyncio/proactor_events.py:804)
recv (asyncio/windows_events.py:493)
_register (asyncio/windows_events.py:720)
and _run_once locals sched_count: 0, ntodo: 1, timeout: None -- nothing is scheduled, one handle re-runs forever.
Related precedent
IocpProactor.recv() in Lib/asyncio/windows_events.py returns b'' in this situation, while recv_into() was changed to return 0 in bpo-41467 -- "asyncio: recv_into() must not return b'' if the socket/pipe is closed". The same reasoning applies to recv(). Independently of that, _loop_self_reading has no EOF handling at all, so a dead self-pipe can never be recovered from.
Reproducer
Deterministic -- 12/12 runs on both interpreters tested.
import asyncio
import socket
import time
async def main():
loop = asyncio.get_running_loop()
# Let the loop arm its self-pipe read first.
await asyncio.sleep(0.1)
# Graceful half-close: the read half sees a clean EOF, which is what an OS
# teardown of the loopback connection looks like.
loop._csock.shutdown(socket.SHUT_WR)
start = time.process_time()
await asyncio.sleep(3)
print(f"CPU consumed while sleeping 3s: {time.process_time() - start:.2f}s")
asyncio.run(main()) # Windows default is ProactorEventLoopExpected: roughly 0.00s -- the process is asleep.
Actual: roughly 2.90s -- a full core burned for a three second sleep.
Important: it must be a graceful half-close. An abortive close() surfaces as an exception, which the existing except BaseException branch handles, so it does not reproduce reliably (~80% of runs, and only when the recv is issued after the socket is already gone).
How this was hit in production
Three unrelated long-running Python programs on the same Windows 11 machine -- two independent MCP servers plus a minimal control program written purely to isolate this -- all began pinning a core at the same instant, roughly 69 minutes into their lifetime. They had accumulated near-identical CPU time (5465s, 5465s, 5485.7s, 5482.7s over ~9638s of uptime), which is what pointed at a shared external trigger rather than three independent bugs.
The trigger turned out to be waking the screen after the machine had been locked -- not going idle. With the display off the machine was silent; moving the mouse lit the screen and every asyncio process immediately pinned a core.
Socket state confirms the mechanism:
| sockets | CPU | |
|---|---|---|
| healthy process | 127.0.0.1:A->B ESTABLISHED + 127.0.0.1:B->A ESTABLISHED (plus vestigial 0.0.0.0:A BOUND) |
0% |
| spinning process | both ESTABLISHED halves gone, only BOUND left |
97-99% |
Nothing is logged, no exception is raised, and the affected processes never recover. From a user's point of view the machine simply starts running hot after every unlock, with several cores pinned, and the only remedy is killing the processes.
The machine has no Modern Standby (powercfg /a reports S3 only), so this is an ordinary session/display transition on a fully awake system, not a sleep/resume cycle.
Environment
- Windows 11
- Python 3.12.10 (
tags/v3.12.10:0cc8128, MSC v.1943 64 bit) -- affected - Python 3.13.12 (
main, MSC v.1944 64 bit) -- affected - The code path is unchanged on
mainas of this writing.
Suggested fix
Treat a zero-length result as the self-pipe being gone, and rebuild it instead of re-arming a read that can never block again:
def _loop_self_reading(self, f=None):
try:
if f is not None:
data = f.result() # may raise
if not data:
# The self-pipe reached EOF: the socketpair is gone (this can
# happen when the OS tears down the loopback connection across a
# power or session state change). Re-arming here would spin the
# CPU forever, so rebuild the pipe instead.
self._self_reading_future = None
self._close_self_pipe()
self._make_self_pipe()
return
...A more conservative variant would be to leave the rebuild out and only stop re-arming, which at least turns an invisible 100% CPU spin into a loop that can no longer be woken -- but rebuilding keeps the loop functional, which seems strictly better.
Making IocpProactor.recv() return a length rather than b'', matching what bpo-41467 did for recv_into(), would additionally make this class of bug harder to reintroduce.
CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Windows
Activity
aidaodedjl commentedon Aug 25, 2026
I can reproduce this deterministically on a self-built main (3.16.0a0): 346,702
_loop_self_readingre-arms during a 3s idle sleep after theSHUT_WRhalf-close, ~1.4s of CPU burned. The empty-result re-arm chain is exactly as analyzed —f.result()returnsb'', no except branch runs,recvon the dead socket completes immediately, forever.One extra data point while digging: the selector twin has the same hole.
SelectorEventLoop._read_from_selfbreaks on the empty read but leaves the reader registered, and a readable-forever closed socket spins the select loop the same way — same reproducer shape againstasyncio.SelectorEventLoopgives 582,692_read_from_selfcalls over 3s idle. I'll file that separately.Fix for the proactor side is up in #156343: rebuild the socketpair on the empty result (new pair allocated before touching the old sockets, wakeup fd re-registered before the old sockets close, next read armed on the new socket). Reproducer drops to 0 re-arms / 0.00s CPU with it, full
test_asynciosuite clean.fix: rebuild the asyncio self-pipe instead of pinning a CPU core
fix: stability batch - self-pipe busy loop, UIA walker, console flash…
pangi commentedon Sep 1, 2026
Sent here from modelcontextprotocol/python-sdk, where this was diagnosed and
closed as upstream. The mechanism in both issues matches what I measured, so I
will not restate it. Four things from the field instead.
1. A trigger that needs no suspend, and takes ten seconds
The reports so far reach the EOF through a suspend/resume cycle. On this machine
the cause turned out to be an in-path WFP filter driver re-applying its
filters — a consumer ad-blocker with a network shield
(
adgnetworkwfpdrv.sys, loaded at System start).Toggling that product's protection off and back on, with nothing else happening:
Both processes, three seconds apart. No sleep involved; system uptime
continuous. Stacks are the ones already in this issue —
_loop_self_reading(proactor_events.py:802) re-arming,_pollreturningimmediately.
Repeated later the same evening with the same result, so two for two on this
machine. What I have not isolated is which of the two WFP transitions does it,
disabling or re-enabling: both deaths land within three seconds of each other,
which means one transition kills both processes, but the test covered both.
It is a clean EOF, not a reset. In the second run the loop was carrying an
instrumented
_loop_self_readingthat reports why it fired, and it saidEOFboth times — never
ConnectionResetError. So this really is the graceful-closepath, which is the one that busy-loops silently rather than raising.
So a class of ordinary consumer software reproduces this on demand.
2. It leaves no trace in any Windows log
This is the part I would flag for severity. At the moment the pair died, nothing
was recorded in System, Application, NetworkProfile, Kernel-PnP or Defender:
Two consequences:
only a pinned core and, if several asyncio processes are running, several
pinned cores at once. Every idle asyncio process on the box goes together,
because they all lose their socketpair in the same instant.
is memorable; "my CPU is at 100% and nothing says why" is not something people
connect to asyncio.
3. What the socket table shows, in case it helps the fix
The connected rows disappear entirely — the pair is not left half-closed:
Same process, same thread, twelve minutes apart. The
Boundremnant keeps theidentical port and the handle count is unchanged, so the interpreter never
closed either fd — the connection was removed underneath it. A synthetic pair
behaves the same way: half-closed states last a couple of seconds and then both
rows are gone while both fds stay open.
If a fix rebuilds the pair on EOF, that is exactly the state it has to recover
from: two live fds whose connection no longer exists in the TCP table.
4. Rebuild-on-EOF holds up outside a test harness
Small vote of confidence for the approach in the open PRs. I am running a
monkeypatched
_loop_self_reading/_read_from_selfthat rebuilds the pair onEOF or error, on both loop types, inside two long-lived stdio servers. With the
trigger above applied:
New pair, new ports, no spin, and the servers kept answering. Before the patch
the identical trigger put both of them at 99-100% of a core within three
seconds.
I also checked the thing a rebuild could plausibly break:
set_wakeup_fdhas tofollow the new
_csock, or cross-thread wakeups go quiet in a way nothingreports.
call_soon_threadsafestill works after the rebuild on both loop types.Worth an assertion in the PR tests if there is not one already.
Environment
Windows 10.0.19045, uv-managed CPython 3.13.14,
mcp1.29.x, two idle stdioservers spawned by the same host application. Full py-spy dumps (four samples
per process,
--native), socket snapshots at detection time, and a healthybaseline of both are available for every episode since 2026-08-29 — say the word
and I will attach them.
pythongh-156333: rebuild the proactor self-pipe on EOF instead of bus…
aidaodedjl commentedon Sep 4, 2026
All four points land, and the WFP trigger is the most useful thing this issue has gotten — a reproducer that needs no suspend cycle is the difference between "we believe the reporter" and "anyone can confirm in ten seconds."
On point 4: the wakeup-fd concern is already covered in both PRs. #156343 re-registers
signal.set_wakeup_fd(csock.fileno())before the old sockets are closed, and the test asserts the new fd; #156345 is stricter — it only migrates when the wakeup fd actually points at the loop's own_csock(a foreign registration is restored untouched), and keeps the old write end alive when the rebuild runs on a worker thread.call_soon_threadsafeafter rebuild is a regression test on both loop types, so nothing goes quietly.The clean-EOF detail in point 1 matters more than it might look: it confirms this is the graceful-close path, which is exactly the one that spins silently instead of raising. A
ConnectionResetErrorwould at least have surfaced somewhere.Yes — please attach the py-spy dumps and socket snapshots. Three episodes of field data beat a synthetic reproducer if a core dev asks how this behaves on real machines.
pythongh-156333: rebuild the proactor self-pipe on EOF instead of bus…
pangi commentedon Sep 4, 2026
Attached: cpython-156333-field-data.zip (32 kB) — py-spy dumps, socket snapshots and monitor traces, grouped into eight directories with a README that says what each one demonstrates.
Short version:
Three days on from 08, that monkeypatch has rebuilt the pair 13 times, every one of them with cause EOF and not a single error case. Over the same three days the watchdog — sampling every five minutes, and with 14 runaways logged before — has recorded none.
One caveat on the 13: I cannot give you individual timestamps. Python block-buffers stderr to a pipe, so those lines only reached the host's log when each process exited, which parks every pair of them at a session boundary regardless of when the rebuild happened. The zero-runaway figure comes from the watchdog's
own timestamped log, so that half is solid. The patch now timestamps and flushes its own output, so anything from here on will be exact.
Notes on the files, so nothing looks odd: the Windows user name is replaced with USER and nothing else was altered; one 2026-08-29 capture is deliberately excluded because it is a synthetic while True: pass decoy I used to test the watchdog, not field data; and 05-...-pid27176 has three of four samples because
the process exited while py-spy was sampling it.
Good to hear the wakeup-fd migration is already handled in both PRs, and that call_soon_threadsafe after rebuild is a regression test on both loop types — that was the one thing I could see going quiet without anyone noticing.
The capture runs automatically now, so if you want a specific measurement taken at the moment the pair dies — socket options, an SO_ERROR read on _ssock, a handle table snapshot — name it and it will be in the next episode.
cpython-156333-field-data.zip
aidaodedjl commentedon Sep 4, 2026
Downloaded and read the whole archive — this is now the evidence set I'd point a core dev at, in this order: 04 (same pid, twelve minutes apart, the TCP table going from one loopback pair Established both ways plus the Bound remnant to only the Bound remnant) is the clearest single exhibit for what rebuild-on-EOF has to recover from, and 01's healthy baseline makes the spin legible at a glance: identical frames,
MainThread (idle)becomes(active+gil).07 is more valuable than a control usually is: the pairs survived a deliberate NIC teardown/rebuild on the same ports, so this is not generic network-stack churn — it is something about the WFP filter re-application pass specifically. That plus 03 (continuous uptime, zero power events) closes off the "it's really just suspend/resume being misdiagnosed" explanation completely.
The stderr block-buffering caveat is good data hygiene to state explicitly, and the watchdog's own timestamped zero-runaway figure is the one that matters anyway — 14 runaways before the patch, zero in the three days after, on the same machine doing the same work.
For the capture rig, two things would be worth having at the moment of death, if the next episode allows:
SO_ERRORgetsockopt on both ends at detection time. If it returns 0 while the TCP rows are gone, that documents why the re-arm loop spins silently — nothing on the socket itself reports an error, only the read can reveal the EOF. That is the strongest argument that a recv-side fix is the only possible interception point.netsh wfp show statecaptured immediately before and after the toggle. That would name the layer, and "filter re-application removes loopback TCP connections" is the kind of sentence that gets this fixed at the right level — or at least gets the issue a footnote in WFP-facing release notes.The 13-rebuilds-all-EOF stat is exactly what the PRs needed: it says the graceful-close path is not an edge case triggered by an unusual test, it is the only path this bug uses in the wild.
pangi commentedon Sep 4, 2026
Correction to my earlier comment: the WFP-toggle reproducer does not hold up, and
I withdraw it. Please don't spend time trying to confirm it as described.
What I tested today
Same machine, same two servers, now with the guard timestamping and flushing its own
output so every self-pipe death has an exact time. Five attempts:
Pairs unchanged throughout, same ports, CPU flat, and the watchdog logged no runaway.
Why I got it wrong
Both original hits landed within seconds of a toggle, and I took that for causation.
What I had not noticed at the time: over the three days since the guard went in, it
has rebuilt the pair 13 times with nobody touching the ad-blocker at all. So the
trigger fires on its own a few times a day. Two coincidences in a row with something
that frequent is unremarkable — and I only ever measured while we were toggling.
So the trigger is not identified. It is something that happens spontaneously a
few times a day on this machine, silently, and I do not yet know what.
What does still stand
None of the following depends on the trigger:
04of the archive:connected loopback rows gone,
Boundremnant on the same port, handle countunchanged,
_loop_self_readingre-arming.03, continuous uptime, zero wakes inpowercfg /lastwake.05.days against 14 in the days before, on the same machine doing the same work.
The SO_ERROR reading you asked for
Captured inside the process at the moment of detection, before the old sockets are
closed:
Zero on both ends, both fds still open, and
getpeername()still resolving onboth — while the connected rows are already gone from the TCP table. Nothing on the
socket reports anything wrong; only the read reveals it. That is your argument for a
recv-side interception point, in the strongest form I can give it.
The WFP diff, and a caveat on the wording
netsh wfp show statebefore the toggle, after disabling, and after re-enabling(attached as
wfp-summaries.zip— the provider's filters and callouts per layer,reduced from three 28 MB dumps).
The three extracts are byte-identical, same SHA-256, apart from the source
filename in the header:
FWPM_LAYER_STREAM_V4/_V6The provider keeps every filter and callout registered whether protection is on or
off. Where it sits, for what it is worth: 2 of the 4 filters at
FWPM_LAYER_STREAM_V4, the same at_V6, 2 of 6 atFWPM_LAYER_ALE_FLOW_ESTABLISHED_V4/V6, both filters atFWPM_LAYER_ALE_ENDPOINT_CLOSURE_V4/V6, and 56Stream Calloutplus 72Flow Established Calloutentries.So "filter re-application removes loopback TCP connections" is not what this
measures — the filter set does not change across the transition. If an in-path
filter is involved at all, it would have to be a callout dropping flows it was
inspecting while its registrations stay put. I would rather hand you the empty diff
than a sentence that reads well and is not supported.
Next
The guard now timestamps and flushes, and captures SO_ERROR on both ends
automatically. The next spontaneous event — a few times a day, going by the count —
will arrive with an exact second and the surrounding event log, and I will post it.
Given the trigger leaves no trace in any Windows log, that may be the only way to
name it.
wfp-summaries.zip
aidaodedjl commentedon Sep 4, 2026
Withdrawing your own reproducer after two public threads credited it is not the usual move — noted and respected. Correcting my own comment above: disregard the "narrowed to the WFP filter re-application pass" reading of 06/07. The toggles were coincidences; the archive's mechanism exhibits (03/04/05, the negative control in 07 as a negative control) all stand on their own.
The empty WFP diff is worth more than a matching one would have been. Same SHA-256 across the transition, every filter and callout still registered: there is no static configuration change to point at, and a callout dropping a flow it inspects would leave exactly this kind of nothing behind. Combined with the 13 unattended rebuilds, the honest summary is: the trigger is spontaneous, silent, unlogged, and possibly never going to be nameable.
Which flips the framing for this issue. Until today the story was "reproduces via WFP toggling, suspend/resume, three-second S3" — multiple causes, and a reviewer could reasonably ask which one is the real target. The corrected story is stronger: the cause does not matter and cannot be relied on.
SO_ERROR=0on both ends withgetpeername()still resolving, while the connected rows are already gone from the TCP table, is the whole argument in one capture — the kernel has torn the connection down, the socket objects have no idea, and the only API surface that can possibly discover the state is the read. There is no earlier interception point and no diagnostic that beats it. Any loop that treats "read returned EOF" as a reason to immediately re-arm the same read is wrong on its face; what the EOF means (peer half-closed? connection destroyed underneath us?) is exactly what the loop cannot know, which is why rebuilding is the only bounded response available.So the pull requests are not waiting on a trigger attribution, and I would argue against deferring them until one arrives — an unnameable trigger is precisely the case where the runtime has to be defensive. If a future capture does name the cause, that becomes a nice README footnote, not a prerequisite.
pythongh-156333: rebuild the proactor self-pipe on EOF instead of bus…
msullivan commentedon Oct 3, 2026
I somewhat lost track of the thread of robots talking to each other in this issue, but is there a real repro procedure (that doesn't involve manually sabotaging event loop internals)?
xksk-doer commentedon Oct 4, 2026
Thanks for the analysis in this issue — we hit the sibling failure mode of the same code line on Windows in production, and it is not covered by the current patch in #156343.
What we see
_loop_self_reading()can also lose the pipe through the exception path, not just EOF. When the pendingrecv()onself._ssockis aborted,f.result()raises instead of returningb'':finish_socket_func()mapsERROR_OPERATION_ABORTED(995) andERROR_NETNAME_DELETED(1236) toConnectionResetError, so this reads like "the peer reset us", but thewinerrorsays it was aborted locally.On main, that path goes to
except BaseException→call_exception_handler(...)→returnwith no re-arm, so the loop keepsrunning=Truewhile its self-pipe is dead. Every subsequentcall_soon_threadsafe()/run_coroutine_threadsafe()from another thread enqueues a callback that is never run, because nothing wakes the loop any more.Note this is strictly worse than the EOF case in this issue: EOF (and #156343's fix) pins a core and is visible; the abort path is silent — one log line, then wakeups are lost forever. It also affects any exception raised there, not only 995.
Reproducer (deterministic, Windows-only, 3.11.15 and 3.14.7)
Output on both interpreters (3/3 runs each):
CancelIoExis only used to inject the abort deterministically; closingloop._ssockfrom another thread produces the same permanent loss (withwinerror=1236). Noteshutdown(SHUT_RD)/shutdown(SHUT_RDWR)on the pipe sockets does not trigger it — it is specific to a cancelled/interrupted overlapped operation.Why #156343 as written doesn't fix it
The patch only rebuilds on the EOF branch:
The
except BaseExceptionbranch is unchanged, so an aborted read still leaves the loop un-wakeable. Would you be open to also rebuilding (or at least re-arming) there? The rebuild helper looks like the right recovery for both cases; if rebuilding on the abort path is considered risky, re-arming at minimum makes the loop recover on the next wakeup instead of staying broken forever.Production evidence (why we chased this)
Long-lived Windows processes that create a private event loop per streaming HTTP request (several hundred loops/hour). In one 4-hour window:
stale_report_time − stale_threshold(i.e. request start) vs the logged error timestamp: delta 0.0 s in all 8 cases.running=True, closed=False, and died 2–20 s after creation.WinError 10054/WSAECONNRESET/RemoteDisconnected: 0 occurrences — the transport was never reset by the remote side.Happy to test a patch on Windows (3.14.7 + 3.11.15) if that helps.
pythongh-156333: recover the proactor self-pipe when the read fails i…
aidaodedjl commentedon Oct 5, 2026
Short answer: no clean one today.
With SO_ERROR reading 0 on both ends at the moment of death, I wouldn't bet on a nameable real-world trigger. The read is the only surface that observes this state. If a CI-able repro matters for the PRs, CancelIoEx is the honest best available.
msullivan commentedon Oct 5, 2026
Merge remote-tracking branch 'origin/main' into pythongh-156333-proac…
aidaodedjl commentedon Oct 5, 2026
Tested it in a Windows 11 guest on Parallels (26200, stock 3.13.13): a 30-second VM suspend/resume with the loop idling left wakeups working, no self-pipe error after resume. The guest's TCP state is part of the snapshot, so the loopback pair comes back exactly as it went in.
Also tried the sharper variant, disconnecting and reconnecting the guest's vNIC from the host while the loop idled: wakeups still fine. Loopback traffic never touches the NIC, and that matches the dir 07 negative control in the archive (a deliberate Hyper-V vSwitch rebuild didn't kill the pairs either).
So suspending a VM reproduces the freeze without the death. Hardware S3 kills (dir 05) because the host reinitializes the network stack under the running process; a VM snapshot just pauses and restores it. A hypervisor setup that reattaches network hardware on resume instead of restoring it might behave differently, but plain suspend on Parallels does not.
msullivan commentedon Oct 5, 2026