Repository navigation
Document fold-wise draws for custom latent functions in K-fold CV - #588
Merged
Merged
Conversation
aloctavodia
approved these changes
Nov 3, 2026
aloctavodia
approved these changes
Nov 4, 2026
aloctavodia
left a comment
Collaborator
There was a problem hiding this comment.
I added a few minor comments.
Co-authored-by: Osvaldo A Martin <aloctavodia@gmail.com>
…m cvrefbuilder and call the replacement a workaround
…nes the custom cvrefbuilder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In K-fold CV, the
cvrefbuilderofget_refmodel.stanreg()andbrms::get_refmodel.brmsfit()passes user-suppliedlatent_ilink,latent_ll_oscale,latent_ppd_oscaleanddisto every fold unchanged. The latent vignette writes the negative binomial and Weibull functions as closures over full-data draws (refm_prec,refm_shape) and passes the full-datasigmaasdisin the log-normal example. A user who switches these examples to K-fold CV (as in https://discourse.mc-stan.org/t/39275, post 4) therefore scores every fold with full-data draws. Since #568, these draws enter the submodel scores.This PR documents the behavior and a remedy, without code changes: define the functions inside a function of the fit that calls
get_refmodel(), and replace elementcvrefbuilderof the reference model object with a function that calls it on the fold refit (get_refmodel.stanreg()andbrms::get_refmodel.brmsfit()do not accept acvrefbuilderargument). It adds a paragraph to?extend_family, a vignette section "K-fold CV with custom response-scale functions" with the negative binomial example, pointers from the Weibull and log-normal examples, and a NEWS bullet.I ran the new vignette chunk on a small negative binomial
stan_glm()model with 2 folds: with the defaultcvrefbuilderboth folds used the full-data precision draws, with the custom one each fold used its own, and the cross-validated elpd changed by 1 to 5 per submodel size. I did not build the full vignette.This PR was assisted by Claude.