Skip to content

fix: avoid overflow and underflow for scales far from 1 in stats/base/dists/rayleigh/pdf - #15994

Open
Abhist17 wants to merge 2 commits into
stdlib-js:developfrom
Abhist17:fix/rayleigh-pdf-scale
Open

Abhist17 wants to merge 2 commits into
stdlib-js:developfrom
Abhist17:fix/rayleigh-pdf-scale

Conversation

@Abhist17

@Abhist17 Abhist17 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Description

What is the purpose of this pull request?

This pull request:

  • avoids overflow and underflow for scale parameters far from 1 in stats/base/dists/rayleigh/pdf (JS, factory and C).

For sigma outside about [1.5e-154, 1.3e154], sigma^2 (or x^2) overflows or underflows, so the PDF comes out NaN or 0 where it is finite:

var pdf = require( '@stdlib/stats/base/dists/rayleigh/pdf' );

pdf( 1.0e200, 1.0e200 );    // develop: NaN   this PR: 6.0653065971263344e-201
pdf( 1.0e-200, 1.0e-200 );  // develop: NaN   this PR: 6.065306597126334e+199
pdf( 1.0e150, 1.0e200 );    // develop: 0     this PR: 1e-250

This is the follow-up I mentioned in #15923, using the same approach. The existing expression is kept wherever it is safe — sigma^2 normal with 2*sigma^2 finite (the 1e154 case you found on #15923), x^2 finite, x/sigma^2 normal — and on all 3,000 points of the existing fixtures the result is bit-for-bit identical to develop. Otherwise it uses x/sigma: (r/sigma) * exp(-r^2/2), falling back to exp( ln(r) - ln(sigma) - r^2/2 ) when x/sigma^2 overflows, i.e. for a tiny sigma, where the exponential decides the result.

Tests, and why the existing fixtures weren't regenerated this time. On #15923 you asked for the old fixtures to be regenerated in extended precision. I tried the same here. Against a 2048-bit reference, develop — and this PR, being bit-identical there — is up to 691 ULP off on the existing fixtures, at x/sigma ≈ 34. That's exp(-r^2/2) amplifying the rounding of x^2/(2*sigma^2) by its own magnitude (~590 at that point), which any double-precision evaluation inherits. The old Float64 Julia fixtures share that rounding, so they pass at 2 ULP. Regenerating them would mean loosening those tolerances to ~700 ULP for a part of the function this PR doesn't change.

So gen and the three existing fixtures are untouched. A separate gen_extended (2048-bit BigFloat, textbook formula) writes two new fixtures:

  • tiny_scale.json: sigma in [1e-200, 1e-155], where sigma^2 underflows.
  • huge_scale.json: sigma in [1e155, 1e200], where it overflows.

Both use x/sigma in [1e-3, 1], so they test the new branch rather than that conditioning. On them this PR is within 2 ULP, the existing tolerance; develop fails all 2,000 points in both test.pdf.js and test.factory.js. test.pdf.js / test.factory.js / test.native.js pass 5,014 each. If you'd still prefer the old fixtures regenerated, I'll do that and set those three blocks' tolerances from the measured errors instead.

Related Issues

Does this pull request have any related issues?

None. Follow-up to #15923 (rayleigh/logpdf), as mentioned there.

Questions

Any questions for reviewers of this pull request?

…e/dists/rayleigh/pdf`

For `sigma` outside about [1.5e-154, 1.3e154], `sigma^2` (or `x^2`)
overflows or underflows, so the PDF came out NaN or 0 where it is finite:
`pdf( 1e200, 1e200 )` returned NaN instead of about 6.065e-201.

Same approach as `rayleigh/logpdf` (stdlib-js#15923): the existing expression is kept
wherever it is safe, bit for bit on every existing fixture point, and the
ratio `x/sigma` is used otherwise.
…/pdf`

Generated by a new `gen_extended` in 2048-bit BigFloat, with `sigma` on
either side of the range where `sigma^2` over- or underflows and
`x/sigma` in [1e-3, 1]. develop fails all 2,000 points; this branch is
within 2 ULP. The existing fixtures and `gen` are unchanged.
@Abhist17
Abhist17 requested a review from a team October 8, 2026 22:55
@stdlib-bot stdlib-bot added Statistics Issue or pull request related to statistical functionality. Needs Review A pull request which needs code review. labels Oct 8, 2026
@stdlib-bot

Copy link
Copy Markdown
Contributor

Hello! Thank you for your contribution to stdlib.

We noticed that the contributing guidelines acknowledgment is missing from your pull request. Here's what you need to do:

  1. Please read our contributing guidelines.

  2. Update your pull request description to include this checked box:

    - [x] Read, understood, and followed the [contributing guidelines](https://github.com/stdlib-js/stdlib/blob/develop/CONTRIBUTING.md)

This acknowledgment confirms that you've read the guidelines, which include:

  • The developer's certificate of origin
  • Your agreement to license your contributions under the project's terms

We can't review or accept contributions without this acknowledgment.

Thank you for your understanding and cooperation. We look forward to reviewing your contribution!

@stdlib-bot

Copy link
Copy Markdown
Contributor

Coverage Report

Package Statements Branches Functions Lines
stats/base/dists/rayleigh/pdf $\\color{red}334/338$
$\\color{green}+98.82\\%$
$\\color{red}39/41$
$\\color{green}+95.12\\%$
$\\color{green}4/4$
$\\color{green}+100.00\\%$
$\\color{red}334/338$
$\\color{green}+98.82\\%$

The above coverage report was generated for the changes in this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs Review A pull request which needs code review. Statistics Issue or pull request related to statistical functionality.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants