Skip to content

fix: send a request a GOAWAY refused again on a fresh connection - #971

Open
lennartschoch wants to merge 2 commits into
benoitc:masterfrom
lennartschoch:fix/h2-retry-refused-request
Open

lennartschoch wants to merge 2 commits into
benoitc:masterfrom
lennartschoch:fix/h2-retry-refused-request

Conversation

@lennartschoch

Copy link
Copy Markdown
Contributor

Follow-up to #961 and #962. Those made a GOAWAY fail only the streams the server refused, and return {error, {goaway, _}} for them. That error is a guarantee: RFC 9113 §6.8 says a stream above last_stream_id was not processed. So far hackney handed it to the caller, who then has to work out that this particular error is safe to retry, and most callers don't. We saw it as payment requests failing a booking when Stripe recycled a connection, with the charge never having happened.

Every other client I looked at retries this itself: Go's http2 transport aborts the stream with an error whose comment reads "indicates that the request should be retried on a new connection", .NET has RetryOnConnectionFailure for exactly "server shutting down (GOAWAY)", the JDK marks the streams above last_stream_id as unprocessed for automatic retry, libcurl sets refused_stream and re-runs the transfer on a fresh connection, and OkHttp recovers through RetryAndFollowUpInterceptor. Go, .NET and curl do it for any method, since the protocol guarantees the request wasn't processed. hackney 1.x did the same kind of transparent reconnect for a stale pooled connection.

This does the same in request/5. When the first attempt comes back as {error, {goaway, _}}, it dials again and sends the request once more, whatever the method. Once only: a refusal from the fresh connection goes to the caller. The one body that can't be sent twice is a fun, which the first attempt may have consumed, so those are returned as before. An unpooled connection the GOAWAY refused is stopped first, since it belongs to the caller who is about to replace it; a pooled one has already left its pool.

Not covered on purpose: send_request/2 on a connection from connect/4, where the caller holds the connection and a retry would have to swap it under them, and a streamed request body refused after its chunks started, which send_body/2 reports and the caller has to redo. Both match what the other clients do.

rebar3 eunit is green, 1183 tests, xref and dialyzer clean. The drain tests that asserted the refusal now assert the retried result, and three new ones cover a refused stream being sent again on a fresh connection, the same for an unpooled connection, and the refusal from the fresh connection reaching the caller rather than looping. All fail without the change. The test server gained a first_stream trigger and a trigger_on option so a server can refuse on every connection.

@benoitc

benoitc commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Thanks, the PR looks OK and pass all tests.

Before to merge, can you update the Error Handling section of guides/http2_guide.md. It still tells callers to handle {goaway, _} themselves. It should say the request is sent again once, and list when the error still reaches the caller. The retry should also apply to HTTP/3 refusals. Please add an h3 test

ALso RST_STREAM(REFUSED_STREAM) is just as safe to resend (§8.7) and currently surfaces as {stream_error, refused_stream} maybe you could look at it as well?

A stream above the GOAWAY's last_stream_id was never processed (RFC 9113
6.8), so request/5 dials again and sends the request once more whatever its
method, as long as the body can be sent twice: a fun body may have been
consumed. Once only: a refusal from the fresh connection reaches the caller.
An unpooled connection the GOAWAY refused is stopped first; a pooled one has
already left its pool.
A GOAWAY is not the only way a server turns a request away unprocessed:
RST_STREAM(REFUSED_STREAM) (RFC 9113 8.7) and H3_REQUEST_REJECTED
(RFC 9114 4.1.1) carry the same guarantee, so request/5 sends those again
as well. They leave the connection serving, so a pooled request goes again
on the same connection.

Two things stood in the way of the HTTP/3 retry:

- checkout_h3 only checked that the connection process was alive. A
  connection draining after a GOAWAY leaves the pool with a cast, which
  the retried request's checkout could overtake, landing it back on the
  draining connection. It now takes the same get_state probe as HTTP/2.
- handle_h3_stream_reset's sync clause was guarded on is_pid(From), which
  a gen_statem From never satisfies, so a reset on a plain HTTP/3 request
  hung the caller until recv_timeout.

The HTTP/2 and HTTP/3 guides describe the retry and list when the refusal
still reaches the caller.
@lennartschoch
lennartschoch force-pushed the fix/h2-retry-refused-request branch from a5d3a13 to 1fe0e44 Compare October 30, 2026 15:54
@lennartschoch

Copy link
Copy Markdown
Contributor Author

Thanks for taking a look! I've addressed all three and rebased on master.

Guide: rewrote the Error Handling section in guides/http2_guide.md so it explains the retry and spells out when you'll still see the error: refused twice, body is a fun, the refusal arrives after request/5 already returned (stream body / async), or you're using send_request/2 on your own connection. Added the same bit to the HTTP/3 guide.

HTTP/3: the retry technically already applied, since h3 returns the same {error, {goaway, _}} — but when I wrote the test it didn't actually work. Two reasons:

  • checkout_h3 only checked the process was alive. The draining connection leaves the pool via a cast, and the retried request's checkout can beat it, so the retry ended up right back on the draining connection. Now uses the same get_state probe as checkout_h2.
  • handle_h3_stream_reset's sync clause was guarded on is_pid(From), but a gen_statem From is a tuple, so a server reset on a plain h3 request just hung until recv_timeout.

Both fixed, with h3 tests for the retry, the second refusal, and H3_REQUEST_REJECTED.

RST_STREAM(REFUSED_STREAM): good shout, done — and I added H3_REQUEST_REJECTED on the h3 side while I was there, since it's the same promise. Both leave the connection usable, so a pooled request just goes again on the same connection.

Unrelated, but worth mentioning: a fun/0 body on HTTP/2 currently crashes the connection in do_h2_send (iolist_to_binary on the fun). I left it alone here — happy to open an issue if you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants