[avehtari]

Closes #566.

This PR was assisted by Claude.

The new function custom_stat() defines a performance statistic from a user function. Argument stats of summary.vsel(), plot.vsel() and performances() accepts the result alone or in a list with built-in statistics. Argument stat of suggest_size() accepts one such object. The statistic changes only the evaluation and the suggested size; the search does not use it.

net_benefit <- function(thres) {
  custom_stat(
    function(y, mu, wobs) (mu >= thres) * (y - (1 - y) * thres / (1 - thres)),
    name = paste0("nb", 100 * thres),
    label = paste0("Net benefit \u2265 ", thres)
  )
}
cvvs <- cv_varsel(fit)
summary(cvvs, stats = list("elpd", net_benefit(0.2)))
plot(cvvs, stats = list("elpd", net_benefit(0.2)))

fun(y, mu, wobs) gets the observed responses, the predictive means and the observation weights of the evaluation set. With cross-validation, mu holds the cross-validated predictive means. For the binomial family, y is the proportion of successes and wobs the number of trials.

Argument type has two values:

• "pointwise" (default): fun returns one value per observation, and the statistic is their weighted mean. The standard error and the difference from the baseline model use the same formulas as "acc". The difference pairs the two models by observation, and subsampled LOO CV uses the difference estimator. Net benefit is a mean of pointwise terms, so it uses this type.
• "aggregate": fun returns one value, for example a PR-AUC. The standard error and the intervals come from the bootstrap, as for "auc", with the same bootstrap samples for both models of a difference. This type does not support subsampled LOO CV.

"acc" and "auc" split a binomial row with several trials into single trials; custom statistics do not. Custom statistics are not available for the augmented-data projection, or for the latent projection with resp_oscale = TRUE and a non-NULL family$cats. In these projections, mu holds one probability per category.

name must contain only letters, digits and underscores, because the summary table adds suffixes such as .se to it. It must not be a built-in name or a column name of the summary table. higher_better sets the direction for baseline = "best" and suggest_size().

The internal parse_stats() turns stats into the names of the statistics and a list of the custom_stat objects. Later code uses the names as before, and get_stat() passes custom statistics to the new get_stat_custom(). The code for the built-in statistics does not change, and stats = NA gives the same error as before.

test_custom_stats.R implements "mse" and "acc" again as pointwise custom statistics and "auc" as an aggregate one. It compares the results with the built-in statistics (tolerance = 1e-10) for the vsel objects of setup.R, including K-fold CV, subsampled LOO CV and the latent projection. It also compares suggest_size() and baseline = "best". The "mse" comparisons leave out these values of the built-in "mse":

• the interval without deltas, which is log-normal;
• the standard error of the difference with subsampled LOO CV, which uses another estimator;
• the standard error for the reference model compared with itself, which is not exactly 0 because of rounding.

Other tests cover the argument checks, the label in the plot facets, binomial data with more than one trial per row, an all-NA reference model (as in a datafit) and the errors for the unsupported projections.

I ran the full test suite with NOT_CRAN=true on CSC Roihu (R 4.6.1, testthat 3.3.2, lme4 2.0.6, brms 2.23.0, rstanarm 2.32.2, cmdstanr 0.9.0.9001, CmdStan 2.39.0):

version	test blocks	expectations passed	failed	skipped	warnings
master f5c9533	273	56,135	0	0	4
this PR	286	56,387	0	0	4

Both runs have the same four warnings: three from optimx about scaling in test_varsel.R, and one lme4 convergence warning in test_datafit.R. The "this PR" row used a commit that differs from this PR only in comments, documentation and the NEWS item. Most new tests need the reference models of setup.R, so they run only with NOT_CRAN=true. The tests of the argument checks also run on CRAN.