Repository navigation
Connection assigned to a request can be closed as expired by another thread (ReadError: [Errno 9] Bad file descriptor) #1132
Replies: 1 comment
|
I can reproduce this on 1.0.9, and it doesn't need luck. The script below holds the two threads at the right points with events, using a wrapper around import http.server, threading, time
import httpcore, httpx
class H(http.server.BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def do_GET(self):
self.send_response(200); self.send_header("Content-Length", "2"); self.end_headers()
self.wfile.write(b"ok")
def log_message(self, *a): pass
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), H)
threading.Thread(target=srv.serve_forever, daemon=True).start()
url = f"http://127.0.0.1:{srv.server_port}/"
ev = {k: threading.Event() for k in "a_assigned go_a b_closing go_b a_active go_a2".split()}
orig = httpcore.ConnectionPool._close_connections
def close_connections(self, closing): # called right after the pool lock is released
me = threading.current_thread().name
if me == "A" and not ev["a_assigned"].is_set(): # A was handed a connection, still IDLE
ev["a_assigned"].set(); ev["go_a"].wait()
if me == "B" and closing: # B picked that same connection as expired, under the lock
ev["b_closing"].set(); ev["go_b"].wait()
orig(self, closing)
httpcore.ConnectionPool._close_connections = close_connections
def trace(name, info): # A: just after IDLE -> ACTIVE, before any socket I/O
if name == "http11.send_request_headers.started":
ev["a_active"].set(); ev["go_a2"].wait()
def get(**kw):
try: print(threading.current_thread().name, client.get(url, **kw).status_code)
except Exception as e: print(threading.current_thread().name, f"{type(e).__name__}: {e}")
client = httpx.Client(limits=httpx.Limits(keepalive_expiry=0.1))
client.get(url) # leaves one idle keep-alive connection in the pool
a = threading.Thread(target=get, name="A", kwargs={"extensions": {"trace": trace}}); a.start()
ev["a_assigned"].wait(); time.sleep(0.2) # keep-alive runs out while A holds the connection
b = threading.Thread(target=get, name="B"); b.start()
ev["b_closing"].wait(); ev["go_a"].set(); ev["a_active"].wait()
ev["go_b"].set(); b.join() # B closes the connection A is now using
ev["go_a2"].set(); a.join()Every run prints this (Windows 11, Python 3.11.9, httpcore 1.0.9, httpx 0.28.1): 10038 (WSAENOTSOCK) is how Windows reports an operation on a closed socket. On Linux and macOS it's EBADF, which matches your The sequence, with links to 1.0.9 (on master the pool differs only on line 294, and
So the stale check is One more thing for anyone on 1.0.9: the surplus-idle check counts all connections rather than idle ones (L294, fixed on master by #1000 but not released). With httpx's default I couldn't confirm the Workarounds that held up in the repro:
I only tested the sync pool. Given there's a deterministic repro, it might be worth a maintainer converting this to an issue so your PR has something to point at. |
Uh oh!
There was an error while loading. Please reload this page.
With several threads sharing one
httpx.Client, a connection that the pool has just assigned to one thread can be closed by another thread, so the first thread fails withhttpcore.ReadError: [Errno 9] Bad file descriptor(sometimesReadTimeout).Cause (httpcore 1.0.9 and current
main)_assign_requests_to_connectionsassigns an idle connection to a request while holding the pool lock. The connection's state is stillIDLE, because it only becomesACTIVEinsideHTTP11Connection.handle_request, which the requesting thread calls later, outside the pool lock._assign_requests_to_connectionsfor its own request. It sees the same connection asIDLE, findshas_expired()true (keep-alive elapsed, or the socket is readable), removes it from the pool and returns it inclosing_connections.handle_requestcan still pass its state check (IDLE->ACTIVE), butHTTP11Connection.close()is then called by the second thread. It setsCLOSEDand closes the socket without taking_state_lock, so the first thread sends or reads on a closed socket.If the close wins the state check, the first thread gets
ConnectionNotAvailableand retries, which is fine. The failure needs the close to land between the state check and the read, so it is rare and only happens with several threads. The same applies to the surplus-idle branch and the "evict an idle connection" branch.Reproduction
Stdlib server plus a shared
httpx.Client, 8 threads, call spacing about the keep-alive expiry:Three 30 s runs on macOS (Python 3.14, httpcore 1.0.9), one unpatched and one patched at a time:
ReadError: [Errno 9] Bad file descriptor, 5 xReadTimeoutReadError: [Errno 9] Bad file descriptor, 3 xReadTimeoutReadError: [Errno 9] Bad file descriptor, 10 xReadTimeoutIt is timing sensitive: with
keepalive_expiryof 0.01 and 0.02 the unpatched runs showed no EBADF. We first saw it in production on Linux ARM64 (AWS Lambda), where it happens about weekly. A single thread cannot hit it, since assignment and sending are then sequential.Suggested fix
Skip connections that are already assigned to a request when cleaning up and when picking an idle connection to evict (
_async, with_syncgenerated byunasync):With this change the same three 30 s runs produce no errors. A deterministic regression test (a connection assigned to a request must not be returned for closing even when expired) fails before the change and passes after, and the existing suite results are unchanged.
One tradeoff: if the server drops an assigned connection before the request is sent, that request now fails like any stale connection does today, instead of the connection being closed under it.
Happy to open a PR with the change, the test and a changelog entry if this looks right.
This was investigated and written with the help of an AI assistant; the reproduction and numbers above come from runs I did.
All reactions