Repository navigation
fix: send a request a GOAWAY refused again on a fresh connection - #971
lennartschoch wants to merge 2 commits into
Conversation
55794b0 to
a5d3a13
Compare
|
Thanks, the PR looks OK and pass all tests. Before to merge, can you update the Error Handling section of guides/http2_guide.md. It still tells callers to handle {goaway, _} themselves. It should say the request is sent again once, and list when the error still reaches the caller. The retry should also apply to HTTP/3 refusals. Please add an h3 test
|
A stream above the GOAWAY's last_stream_id was never processed (RFC 9113 6.8), so request/5 dials again and sends the request once more whatever its method, as long as the body can be sent twice: a fun body may have been consumed. Once only: a refusal from the fresh connection reaches the caller. An unpooled connection the GOAWAY refused is stopped first; a pooled one has already left its pool.
A GOAWAY is not the only way a server turns a request away unprocessed: RST_STREAM(REFUSED_STREAM) (RFC 9113 8.7) and H3_REQUEST_REJECTED (RFC 9114 4.1.1) carry the same guarantee, so request/5 sends those again as well. They leave the connection serving, so a pooled request goes again on the same connection. Two things stood in the way of the HTTP/3 retry: - checkout_h3 only checked that the connection process was alive. A connection draining after a GOAWAY leaves the pool with a cast, which the retried request's checkout could overtake, landing it back on the draining connection. It now takes the same get_state probe as HTTP/2. - handle_h3_stream_reset's sync clause was guarded on is_pid(From), which a gen_statem From never satisfies, so a reset on a plain HTTP/3 request hung the caller until recv_timeout. The HTTP/2 and HTTP/3 guides describe the retry and list when the refusal still reaches the caller.
a5d3a13 to
1fe0e44
Compare
|
Thanks for taking a look! I've addressed all three and rebased on master. Guide: rewrote the Error Handling section in HTTP/3: the retry technically already applied, since h3 returns the same
Both fixed, with h3 tests for the retry, the second refusal, and
Unrelated, but worth mentioning: a |
Follow-up to #961 and #962. Those made a GOAWAY fail only the streams the server refused, and return
{error, {goaway, _}}for them. That error is a guarantee: RFC 9113 §6.8 says a stream abovelast_stream_idwas not processed. So far hackney handed it to the caller, who then has to work out that this particular error is safe to retry, and most callers don't. We saw it as payment requests failing a booking when Stripe recycled a connection, with the charge never having happened.Every other client I looked at retries this itself: Go's http2 transport aborts the stream with an error whose comment reads "indicates that the request should be retried on a new connection", .NET has
RetryOnConnectionFailurefor exactly "server shutting down (GOAWAY)", the JDK marks the streams abovelast_stream_idas unprocessed for automatic retry, libcurl setsrefused_streamand re-runs the transfer on a fresh connection, and OkHttp recovers throughRetryAndFollowUpInterceptor. Go, .NET and curl do it for any method, since the protocol guarantees the request wasn't processed. hackney 1.x did the same kind of transparent reconnect for a stale pooled connection.This does the same in
request/5. When the first attempt comes back as{error, {goaway, _}}, it dials again and sends the request once more, whatever the method. Once only: a refusal from the fresh connection goes to the caller. The one body that can't be sent twice is a fun, which the first attempt may have consumed, so those are returned as before. An unpooled connection the GOAWAY refused is stopped first, since it belongs to the caller who is about to replace it; a pooled one has already left its pool.Not covered on purpose:
send_request/2on a connection fromconnect/4, where the caller holds the connection and a retry would have to swap it under them, and a streamed request body refused after its chunks started, whichsend_body/2reports and the caller has to redo. Both match what the other clients do.rebar3 eunitis green, 1183 tests, xref and dialyzer clean. The drain tests that asserted the refusal now assert the retried result, and three new ones cover a refused stream being sent again on a fresh connection, the same for an unpooled connection, and the refusal from the fresh connection reaching the caller rather than looping. All fail without the change. The test server gained afirst_streamtrigger and atrigger_onoption so a server can refuse on every connection.