Skip to content

Fix full-data dis in K-fold reference models of get_refmodel.stanreg() - #587

Merged
avehtari merged 4 commits into
masterfrom
kfold-fold-dis
Nov 6, 2026
Merged

avehtari merged 4 commits into
masterfrom
kfold-fold-dis

Conversation

@avehtari

@avehtari avehtari commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

In get_refmodel.stanreg(), cvrefbuilder passes dis = dis, and the function sets dis to the full-data sigma draws only after defining cvrefbuilder. R evaluates dis when cvrefbuilder runs, so in cv_varsel(cv_method = "kfold") every fold-wise reference model of a Gaussian model used the full-data sigma instead of the sigma of its own refit. The fix passes the user-supplied dis (usually NULL), so each fold takes sigma from its refit. The new test in test_refmodel.R fails without the fix.

Effect, measured on Gaussian stan_glm() models with 10-fold CV (N 30 to 400, P 5 and 25, 100 simulated data sets per cell): the old code overestimated the reference model's elpd by about 1 for every N. For N ≥ 100, it overestimated the submodels' elpd by 0.15 to 1, growing with the submodel size; that is at most 0.15 SE on average, and up to 0.7 SE for N 30. The rankings do not change, but the suggested size changed in up to 13 % of the data sets.

brms has the same pattern in get_refmodel.brmsfit(), which needs a separate PR.

This PR was assisted by Claude

@avehtari avehtari added the bug Bugs. label Oct 28, 2026
@avehtari
avehtari requested a review from aloctavodia October 28, 2026 07:50
@avehtari
avehtari merged commit ce76d63 into master Nov 6, 2026
4 of 5 checks passed
@avehtari
avehtari deleted the kfold-fold-dis branch November 6, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bugs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants