[sferik]

This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in duel/ against main and the branch side by side, and fails the build if the branch is slower or allocates more.

Benchmarking an HTTP client is tricky because sockets are too noisy to time. So most of these 31 benchmarks pass a small in-memory socket through the existing socket_class option, which serves a fixed response. That way a whole request (building it, writing it, parsing the response with llhttp, and reading the body) runs without making any system calls, so it can be timed reliably. There's only one benchmark over a real loopback connection, and Duel only checks its allocations.

The rest cover URI parsing and normalization, building and merging headers, building and writing requests, multipart encoding, response parsing, and chaining options. Large bodies turned out to be too noisy, because that time is mostly spent copying memory, so those benchmarks use 8 to 128 KB bodies.

To check that it actually catches things, I added a .dup to Request::Writer#join_headers, and Duel failed 11 benchmarks. Extra work in the parser's on_header_field callback failed 10, and the profile in the report pointed straight at that method. Comparing main with itself passed three times in a row.

I didn't benchmark TLS, IDNs, or cookies, since that time is spent in OpenSSL, Addressable, and http-cookie rather than in this gem.

Duel requires Ruby 3.4, so the Gemfile only adds it on 3.4 and newer, and the Rake task is skipped when it isn't installed. Everything else still works on older Rubies.

After a merge, each push to main is compared with the last commit Duel passed, which the workflow marks with a duel/passed/main branch, so small slowdowns add up until one is caught. A regression in a push doesn't fail the run. Instead, the workflow comments on the commit with the report. A pull request that's slower on purpose can pass with a duel: accept label (the workflow needs re-running after it's added).

[github-actions]

Duel passed: none of 31 benchmarks slower

• Baseline: cdc0d63, where HEAD left origin/main
• Candidate: 2522bff

31 benchmarks in 13s. A timing is slower when a 99.0% confidence interval for its ratio, the candidate's to the baseline's, is above 1.02.

Benchmark	Verdict	Timings (ratio [interval])	Counts an iteration
request: encode a multipart upload of 16 KB	unsettled	cpu 1.165 [1.008, 1.403] undecided

Across the board

• cpu 0.998 [0.994, 1.003] same (limit 1.0050, 30 benchmarks)

Unsettled

A timing too close to the limit to settle doesn't fail the run, but one that's unsettled in run after run is noisy: request: encode a multipart upload of 16 KB.

30 benchmarks unchanged

Benchmark	Timings (ratio [interval])
client: GET a JSON response	cpu 0.994 [0.963, 1.007]
client: POST JSON and parse the reply	cpu 0.979 [0.739, 1.008]
client: stream a 128 KB body	cpu 1.005 [0.982, 1.018]
client: GET a chunked body	cpu 1.004 [0.993, 1.020]
client: GET on a persistent connection	cpu 1.002 [0.799, 1.012]
client: GET a gzipped body with auto_inflate	cpu 1.013 [0.991, 1.018]
client: follow a redirect	cpu 0.993 [0.974, 1.005]
client: GET over a loopback TCP connection
headers: build 5 from a hash	cpu 1.001 [0.979, 1.013]
headers: build 50 from a hash	cpu 1.004 [0.994, 1.014]
headers: look up a response's headers	cpu 1.002 [0.992, 1.016]
headers: merge request headers	cpu 0.987 [0.880, 1.008]
options: chain headers, auth, accept, timeout	cpu 0.994 [0.982, 1.006]
options: merge a request's options	cpu 0.995 [0.522, 1.014]
request: build a GET	cpu 1.000 [0.984, 1.012]
request: build a GET with query params	cpu 1.000 [0.991, 1.017]
request: build a POST with a JSON body	cpu 1.003 [0.996, 1.015]
request: build a POST with a form	cpu 0.993 [0.969, 1.019]
request: write a GET	cpu 0.989 [0.972, 1.015]
request: write a POST with an 8 KB body	cpu 0.997 [0.987, 1.007]
request: write a chunked body of 16 KB	cpu 0.994 [0.980, 1.011]
response: parse a small response	cpu 0.994 [0.981, 1.016]
response: parse 30 headers and a JSON body	cpu 0.995 [0.986, 1.004]
response: parse a chunked body of 16 KB	cpu 0.998 [0.983, 1.014]
response: build one and read its metadata	cpu 1.003 [0.994, 1.015]
uri: parse a typical URL	cpu 1.008 [0.994, 1.016]
uri: normalize a typical URL	cpu 0.990 [0.884, 1.017]
uri: normalize case, port and dot segments	cpu 0.998 [0.963, 1.010]
uri: normalize a URL with a 2 KB query	cpu 0.988 [0.980, 0.998]
uri: join a relative redirect	cpu 1.005 [0.986, 1.019]

[sferik]

I updated the Perf workflow here after a review of the same workflow in another project. The Duel job now only reads, and its checkout keeps no credentials. A separate Comment job, which checks nothing out and runs none of the branch's code, posts the report. .duel.rb also watches duel/, .duel.rb and the workflow, so a pull request that changes only the benchmarks is compared too.

[sferik]

I updated this to Duel 0.1.1. The Perf workflow now only comments when Duel fails, posts a new comment instead of editing the old one, and the report is much shorter.

[github-actions]

🟢 Duel: 100% of benchmarks passed

Compared e7a320e with cdc0d63 (where HEAD left origin/main).

Details: 31 benchmarks in 24s, 5 unsettled

Benchmark	Verdict	Timings (ratio [interval])
options: merge a request's options	unsettled	cpu 1.009 [0.985, 1.023] undecided
request: write a GET	unsettled	cpu 1.005 [0.997, 1.032] undecided
request: encode a multipart upload of 16 KB	unsettled	cpu 1.011 [0.988, 1.036] undecided
response: build one and read its metadata	unsettled	cpu 1.009 [1.001, 1.024] undecided
uri: parse a typical URL	unsettled	cpu 1.009 [0.982, 1.029] undecided
client: GET a JSON response	same	cpu 0.989 [0.968, 1.007]
client: POST JSON and parse the reply	same	cpu 0.993 [0.972, 1.005]
client: stream a 128 KB body	same	cpu 0.995 [0.980, 1.012]
client: GET a chunked body	same	cpu 1.005 [0.994, 1.017]
client: GET on a persistent connection	same	cpu 0.996 [0.979, 1.017]
client: GET a gzipped body with auto_inflate	same	cpu 0.985 [0.962, 1.000]
client: follow a redirect	same	cpu 0.994 [0.973, 1.017]
client: GET over a loopback TCP connection	same
headers: build 5 from a hash	same	cpu 1.000 [0.983, 1.014]
headers: build 50 from a hash	same	cpu 0.994 [0.987, 1.008]
headers: look up a response's headers	same	cpu 1.003 [0.986, 1.017]
headers: merge request headers	same	cpu 1.005 [0.990, 1.019]
options: chain headers, auth, accept, timeout	same	cpu 0.978 [0.949, 1.004]
request: build a GET	same	cpu 0.998 [0.972, 1.009]
request: build a GET with query params	same	cpu 0.991 [0.803, 1.016]
request: build a POST with a JSON body	same	cpu 0.983 [0.961, 1.013]
request: build a POST with a form	same	cpu 0.998 [0.961, 1.020]
request: write a POST with an 8 KB body	same	cpu 0.988 [0.960, 1.012]
request: write a chunked body of 16 KB	same	cpu 0.975 [0.953, 0.998]
response: parse a small response	same	cpu 0.995 [0.987, 1.010]
response: parse 30 headers and a JSON body	same	cpu 0.994 [0.966, 1.009]
response: parse a chunked body of 16 KB	same	cpu 0.976 [0.955, 1.010]
uri: normalize a typical URL	same	cpu 0.997 [0.981, 1.018]
uri: normalize case, port and dot segments	same	cpu 1.001 [0.986, 1.012]
uri: normalize a URL with a 2 KB query	same	cpu 1.009 [0.976, 1.016]
uri: join a relative redirect	same	cpu 1.002 [0.973, 1.016]

An unsettled timing was too close to the limit to settle. It doesn't fail the run, but one that's unsettled in run after run is noisy.

Across the board:

• cpu 0.997 [0.991, 1.005] same (limit 1.0050, 30 benchmarks)

A timing is slower when a 99.0% confidence interval for its ratio, the candidate's to the baseline's, is above 1.02. How Duel decides