Skip to content

Document fold-wise draws for custom latent functions in K-fold CV - #588

Merged
avehtari merged 6 commits into
masterfrom
latent-kfold-cvrefbuilder
Nov 5, 2026
Merged

avehtari merged 6 commits into
masterfrom
latent-kfold-cvrefbuilder

Conversation

@avehtari

Copy link
Copy Markdown
Member

In K-fold CV, the cvrefbuilder of get_refmodel.stanreg() and brms::get_refmodel.brmsfit() passes user-supplied latent_ilink, latent_ll_oscale, latent_ppd_oscale and dis to every fold unchanged. The latent vignette writes the negative binomial and Weibull functions as closures over full-data draws (refm_prec, refm_shape) and passes the full-data sigma as dis in the log-normal example. A user who switches these examples to K-fold CV (as in https://discourse.mc-stan.org/t/39275, post 4) therefore scores every fold with full-data draws. Since #568, these draws enter the submodel scores.

This PR documents the behavior and a remedy, without code changes: define the functions inside a function of the fit that calls get_refmodel(), and replace element cvrefbuilder of the reference model object with a function that calls it on the fold refit (get_refmodel.stanreg() and brms::get_refmodel.brmsfit() do not accept a cvrefbuilder argument). It adds a paragraph to ?extend_family, a vignette section "K-fold CV with custom response-scale functions" with the negative binomial example, pointers from the Weibull and log-normal examples, and a NEWS bullet.

I ran the new vignette chunk on a small negative binomial stan_glm() model with 2 folds: with the default cvrefbuilder both folds used the full-data precision draws, with the custom one each fold used its own, and the cross-validated elpd changed by 1 to 5 per submodel size. I did not build the full vignette.

This PR was assisted by Claude.

@avehtari avehtari added the documentation Documentation issues. label Oct 28, 2026
@avehtari
avehtari requested a review from aloctavodia October 28, 2026 08:02
Comment thread vignettes/latent.Rmd Resolved
Comment thread vignettes/latent.Rmd Resolved
Comment thread vignettes/latent.Rmd Resolved

@aloctavodia aloctavodia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few minor comments.

@avehtari
avehtari merged commit 7da1b5a into master Nov 5, 2026
3 of 5 checks passed
@avehtari
avehtari deleted the latent-kfold-cvrefbuilder branch November 5, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Documentation issues.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants