Skip to content

Add a performance benchmark for forward mode autodiff - #1664

Merged
mborland merged 6 commits into
boostorg:developfrom
reach2sayan:feature/autodiff-performance-benchmark
Nov 21, 2026
Merged

mborland merged 6 commits into
boostorg:developfrom
reach2sayan:feature/autodiff-performance-benchmark

Conversation

@reach2sayan

@reach2sayan reach2sayan commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Adds reporting/performance/autodiff_performance.cpp, a Google Benchmark for fvar (forward mode autodiff).

  • Horner: a degree-63 polynomial
  • Transcendental: exp(sin(x)) * log(1 + x²) / sqrt(x)
  • BlackScholes: the call price from example/autodiff_black_scholes.cpp (log, erfc)
  • MseGibbsEnergy: the excess Gibbs energy of a 16-species mixed-solvent electrolyte as a function of pressure (exp, log, sqrt, pow, and a double loop of mixed fvar/double arithmetic)

Proposition

Differentiating $y = \exp(x)$ with respect to $\varepsilon$ gives $y' = x' y$. Expanding both sides as power series gives

$$\sum_{k=1}^{N} k y_k \varepsilon^{k-1} = \left(\sum_{m=1}^{N} m x_m \varepsilon^{m-1}\right) \left(\sum_{j=0}^{N} y_j \varepsilon^{j}\right)$$

The terms contributing to $\varepsilon^{k-1}$ are those with $j = k - m$. Equating the coefficients of $\varepsilon^{k-1}$ on both sides therefore yields

$$ y_0 = e^{x_0} $$ $$ y_k = \frac{1}{k} \sum_{m=1}^{k} m x_m y_{k-m} \quad \text{for} \quad 1 \le k \le N $$

Since each $y_k$ depends only on $y_0, \dots, y_{k-1}$, the coefficients are computed in increasing order of $k$, and the $k$-th one costs $k$ multiplications. The total cost
is therefore

$$ \sum_{k=1}^{N} k = \frac{N(N+1)}{2} = \Theta(N^2) $$

multiplications and a single scalar exp. The present implementation, by comparison, evaluates $$\exp(x) = \sum_{i=0}^{N} \frac{e^{x_0}}{i!} (x - x_0)^i$$ by Horner's scheme. This performs $N$ full series products of $\Theta(N^2)$ multiplications each, for a total of $\Theta(N^3)$.

@NAThompson Would this be of interest? Otherwise I guess a benchmark is good enough

@NAThompson

Copy link
Copy Markdown
Collaborator

@reach2sayan : It is indeed of interest. Could you post the benchmark results on this thread?

@reach2sayan

Copy link
Copy Markdown
Contributor Author

Google Benchmark's complexity fit over all orders 1–32:

Results on an AMD Ryzen 7 PRO 6850U, GCC 15.2, -O2, pinned to one core, median of 5 repetitions: N = derivative order,

Workload N=1 N=2 N=4 N=8 N=16 N=32 Fit
Horner 417 ns 681 ns 901 ns 1.48 µs 3.85 µs 13.3 µs N²
Transcendental 75 ns 110 ns 246 ns 840 ns 3.72 µs 24.2 µs N³
BlackScholes 64 ns 160 ns 306 ns 902 ns 4.29 µs 29.5 µs N³
MseGibbsEnergy 329 ns 1.64 µs 2.12 µs 3.59 µs 8.72 µs 42.2 µs N²

@NAThompson

Copy link
Copy Markdown
Collaborator

@reach2sayan : If you don't mind I'm going to try to get this fixed in this PR-I see you "allow edit by maintainers".

@NAThompson

Copy link
Copy Markdown
Collaborator

@reach2sayan: you are right. I pushed two commits to your branch: multivariate workloads for the benchmark, then the recurrences for exp, log, sqrt, pow, sin, cos, sinh, cosh, tan, the inverse trigonometric and hyperbolic functions, erf and erfc.

Apple M-series, clang -O2, time per evaluation before → after.

One variable, by derivative order:

Workload 1 2 4 8 16 32
Horner 119 → 117 ns 120 → 122 ns 154 → 150 ns 247 → 246 ns 3.77 → 3.73 µs 14.0 → 13.7 µs
Transcendental 41 → 32 ns 86 → 46 ns 363 → 264 ns 4.18 → 1.36 µs 26.5 → 4.60 µs
BlackScholes 32 → 31 ns 47 → 38 ns 121 → 63 ns 373 → 208 ns 4.64 → 1.20 µs 31.6 → 3.79 µs
MseGibbsEnergy 273 → 276 ns 300 → 291 ns 511 → 492 ns 1.09 µs → 981 ns 5.28 → 2.52 µs 33.3 → 11.1 µs

Several variables, by order in each variable:

Workload 1 2 3 4 6 12
TwoVariables 84 → 35 ns 266 → 72 ns 656 → 195 ns 1.64 µs → 401 ns 6.75 → 1.03 µs 249 → 28.6 µs
ThreeVariables 97 → 63 ns 447 → 171 ns 1.21 µs → 569 ns 2.88 → 1.15 µs 10.8 → 3.69 µs
BlackScholesFourVariables 304 → 213 ns 4.26 → 2.07 µs 61.4 → 16.2 µs 287 → 48.2 µs

Against 100-digit arithmetic, the worst relative error in the 33 coefficients of Transcendental at order 32 goes from 2e8 ε to 14 ε. One tolerance in test_autodiff_2.cpp moves from 1 ε to 4 ε.

@reach2sayan

reach2sayan commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@NAThompson I built a bit more on your commits.

  1. log solves $x y' = x'$ directly. The previous design built the full series of $1/x$ and then integrated it, which is two $N^2$ passes. The new integrate_log needs one:
    $$k x_0 y_k = k x_k - \sum_{j=1}^{k-1} (k-j), x_j, y_{k-j}$$

  2. integrate_pow divides once hence (sum /= k) / x0 became sum / (k * x0).

  3. ode_coefficient multiplied by $j$ inside every term, and always looped all the way to $k$. The new dx(n) computes $d_j = j x_j$ once, along with $n$, the index of the last nonzero $d_j$. ode_coefficient then just sums $\sum_{j=1}^{\min(k,n)} d_j, g_{k-j}$.

  4. The inverse trigonometric and hyperbolic functions use pow(w, -0.5) : asin, acos, asinh and acosh built their derivative as sqrt(w).inverse(): a sqrt recurrence followed by a full series division. pow(w, -0.5) is a single integrate_pow pass. Also fixes a NaN. At asin(1), $w = 1 - x^2$ has $w_0 = 0$, and sqrt(w) is [0, -inf, -inf, ...]. Inverting that series computes inf - inf and gives NaN from the third derivative on. At $w_0 = 0$, pow falls back to the old code, which computes each derivative directly. asin(1) now returns $+\infty$ for every derivative and acos(-1) alternates $-\infty, +\infty$.

Should be faster. Could you confirm.

@reach2sayan

Copy link
Copy Markdown
Contributor Author

@NAThompson Also I did clock the ulp failure. I get what ulp is but I do not have the appreciation / understanding of how that can bite. Could you be the judge of that. Perhaps briefly give me some knowledge given it's within context right now

@NAThompson

NAThompson commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Tests pass on 3ee55bb. Tables below compare develop, 8766c50, 3ee55bb and a proposed follow-up (Apple M-series, clang -O2).

Proposed is 3ee55bb without the dx precompute in ode_coefficient, and with 1 / sqrt(x0) in place of pow(x0, -0.5) as the leading coefficient for asin, acos, asinh and acosh. InverseFunctions is a new workload, asin(x) * acosh(1 + x) + acos(x) * asinh(x).

Time per evaluation:

Workload Order develop 8766c50 3ee55bb proposed
Transcendental 2 40 ns 32 ns 34 ns 32 ns
Transcendental 4 86 ns 48 ns 70 ns 46 ns
Transcendental 8 356 ns 252 ns 217 ns 241 ns
Transcendental 16 4.09 µs 1.33 µs 1.10 µs 1.18 µs
Transcendental 32 26.2 µs 4.38 µs 3.84 µs 3.92 µs
InverseFunctions 2 54 ns 52 ns 100 ns 44 ns
InverseFunctions 4 150 ns 118 ns 148 ns 80 ns
InverseFunctions 8 649 ns 448 ns 456 ns 308 ns
InverseFunctions 16 8.03 µs 2.63 µs 1.56 µs 1.60 µs
InverseFunctions 32 53.7 µs 8.96 µs 4.62 µs 5.13 µs
BlackScholes 2 46 ns 38 ns 42 ns 37 ns
BlackScholes 4 122 ns 68 ns 105 ns 67 ns
BlackScholes 8 371 ns 296 ns 331 ns 280 ns
BlackScholes 16 4.59 µs 1.24 µs 1.13 µs 1.16 µs
BlackScholes 32 31.4 µs 4.15 µs 3.71 µs 3.81 µs
MseGibbsEnergy 2 300 ns 296 ns 297 ns 294 ns
MseGibbsEnergy 4 510 ns 503 ns 482 ns 484 ns
MseGibbsEnergy 8 1.09 µs 987 ns 930 ns 925 ns
MseGibbsEnergy 16 5.37 µs 2.50 µs 2.18 µs 2.28 µs
MseGibbsEnergy 32 33.4 µs 10.9 µs 10.1 µs 10.2 µs
TwoVariables 1 86 ns 37 ns 36 ns 35 ns
TwoVariables 2 265 ns 81 ns 75 ns 70 ns
TwoVariables 4 1.62 µs 414 ns 362 ns 363 ns
TwoVariables 8 22.6 µs 4.15 µs 3.85 µs 3.89 µs
TwoVariables 12 251 µs 26.9 µs 24.0 µs 24.1 µs
ThreeVariables 1 95 ns 61 ns 58 ns 61 ns
ThreeVariables 2 449 ns 175 ns 188 ns 170 ns
ThreeVariables 4 2.88 µs 1.17 µs 1.20 µs 1.18 µs
ThreeVariables 6 10.8 µs 3.70 µs 3.54 µs 3.55 µs
BlackScholesFourVariables 1 304 ns 266 ns 229 ns 264 ns
BlackScholesFourVariables 2 4.28 µs 2.12 µs 2.24 µs 2.11 µs
BlackScholesFourVariables 3 61.2 µs 15.7 µs 15.4 µs 15.7 µs
BlackScholesFourVariables 4 266 µs 47.7 µs 46.7 µs 47.3 µs

Worst relative error of a Taylor coefficient, in ε, orders 0 to 8 (200 random arguments, one variable, against 100-digit arithmetic):

Function develop 8766c50 3ee55bb proposed
exp 0.82 1.3 1.3 1.3
log 2.1 2.1 2.2 2.2
sqrt 2.2 2.5 2.7 2.7
pow(x, -1.5) 2.9 3.2 3.1 3.1
pow(2, x) 2.8 2.9 2.9 2.9
sin 0.81 1.2 1.2 1.2
cos 0.81 1.2 1.2 1.2
sinh 0.76 1.3 1.3 1.3
cosh 0.79 1.3 1.3 1.3
tan 15 15 15 15
asin 10 8 8.8 8.8
acos 10 8 8.8 8.8
atan 511 511 511 511
asinh 6.2e4 5.7e4 4.9e4 8.4e4
acosh 14 14 9.8 10
atanh 7.7 7.7 7.7 7.7
erf 309 179 179 179
erfc 309 179 179 179
tanh 6.0e4 6.0e4 6.0e4 6.0e4
Transcendental 2445 1.0e4 7803 7803

Worst relative error of a Taylor coefficient, in ε, orders 9 to 32 (200 random arguments, one variable, against 100-digit arithmetic):

Function develop 8766c50 3ee55bb proposed
exp 1 2.7 2.7 2.7
log 4.1 4.1 6.2 6.2
sqrt 4.5 5.4 7.2 7.2
pow(x, -1.5) 5.8 6.6 6.3 6.3
pow(2, x) 8.2 10 10 10
sin 1 3.6 3.6 3.6
cos 1 3.6 3.6 3.6
sinh 1.1 4.2 4.2 4.2
cosh 1 4.2 4.2 4.2
tan 210 210 210 210
asin 38 33 37 37
acos 38 33 37 37
atan 2.5e4 2.5e4 2.5e4 2.5e4
asinh 1.7e9 4920 3332 3332
acosh 4.3e6 60 45 39
atanh 31 31 31 31
erf 2.4e6 601 601 601
erfc 2.4e6 601 601 601
tanh 2.1e6 9.4e5 9.4e5 9.4e5
Transcendental 1.8e13 1.0e6 2.6e6 2.6e6

@NAThompson

Copy link
Copy Markdown
Collaborator

Perhaps briefly give me some knowledge given it's within context right now

Yeah that's a judgement call for sure. I think that this is an obvious improvement but there might be some isolated tests that fail.

@reach2sayan

Copy link
Copy Markdown
Contributor Author

@NAThompson Look the best so far. Certainly an improvement. So we are all set on this PR?

reach2sayan and others added 6 commits November 19, 2026 14:06
Co-Authored-By: Claude <noreply@anthropic.com>
O(N^2) multiplications at order N, not the O(N^3) of Horner.

Co-Authored-By: Claude <noreply@anthropic.com>
Skip zero coefficients; integer powers and non-finite roots
take the old path; sin and cos share one recursion.

Co-Authored-By: Claude <noreply@anthropic.com>
Drop the dx precompute; seed asin, acos, asinh, acosh with
1/sqrt, not pow. Benchmark the inverse functions.

Co-Authored-By: Claude <noreply@anthropic.com>
@reach2sayan
reach2sayan force-pushed the feature/autodiff-performance-benchmark branch from 8ba79a7 to 6f54f80 Compare November 19, 2026 19:10
@mborland
mborland merged commit 9ecd4f9 into boostorg:develop Nov 21, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants