Repository navigation
Test which doses differ from the vehicle with multiple contrast tests - #82
Merged
parashardhapola merged 16 commits intoNov 20, 2026
Merged
Conversation
Dunnett's and Williams-type tests refer the largest of correlated t-statistics to the multivariate t distribution, which CytePool could not compute. mvtnorm's default absolute error of 1e-3 cannot resolve the p-values that calls across 20,000 genes need, whose Benjamini-Hochberg cut-off for the top gene is 2.5e-6. multivariate_t_cdf computes the probability of a box and its complement, and multivariate_t_quantile the critical value of a max-t test, each with its standard error, at any df, integer or not. The complement is a sum of first exceedances, each conditioned on its exceedance and integrated with 16 randomly shifted lattices, so that the standard error stays at most relative_error (1% by default) of the smaller of the probability and the complement, down to the smallest doubles. Two coordinates are integrated over directions by adaptive quadrature, without random numbers; singular correlations, infinite limits and NaN are handled.
A dose series asks which doses differ from the vehicle in each gene. Bonferroni's correction ignores the correlation that the shared vehicle creates, and an F-test says whether any dose differs, not which. multiple_contrast_test runs single-step multiple contrast tests over every gene from estimates and their covariances, as multcomp's glht does with parm(): estimates, standard errors, t-statistics and p-values adjusted for the gene's contrasts, with a table of each gene's max-t test and its Benjamini-Hochberg adjustment over genes. dunnett_contrasts and williams_contrasts build each dose against the vehicle, and the vehicle against the size-weighted means of the highest doses, as Contrasts matched to the estimates by name.
validation/multiple_contrast_tests.py compares the probabilities with Ding's conditional distribution and the average over directions in mpmath, Dunnett's integral, mvtnorm and scipy; the quantiles with their roots, with mvtnorm and by a round trip; and the contrast matrices and tests with multcomp, whose p-values it compares where its absolute error of 1e-3 resolves them. Its mpmath references divide each integrand by its largest value at the breakpoints first, as mpmath's quadrature stops at an absolute error of 1e-30 and left orthants of 1e-51 and 1e-87 with 9 correct digits. The tests read the fixtures the script writes, besides closed forms, Dunnett's integral computed in the test and invariants of the tests, and check that the stated errors are honest over hundreds of seeds.
"Multiple testing" gains a section on several contrasts of one gene: the max-t test, single-step p-values against Bonferroni's, Dunnett's and Williams-type contrasts, how the probabilities are computed and why their error is relative. The bulk RNA-seq guide tests both drugs against the control at one error rate per gene and compares the genes it calls with the F-test's. The Validation page gains the section of multiple_contrast_tests.py, its summary rows, its benchmark and its commands, and records its run; "Choosing a method", the glossary and the bibliography follow.
The data of the other tests give p-values in order without it, so removing the running maximum went unnoticed. The test feeds it a less extreme contrast whose estimate fell below a more extreme one's, and checks that it takes that p-value and its error.
…inate Each round of the quantile search narrowed its bracket around the previous root and needed two loops to widen it again when more points moved the root outside; no test reached those loops, and five calls repeated sixteen arguments. Nothing compared limits that differ between coordinates with a reference, so a part's limits taken in the wrong order passed every test. Each round now bisects again between the univariate quantile and Bonferroni's, which always hold the root, and clamps the interpolated root to them; the slope comes from the final bracket, and the parts and fixed arguments travel as tuples. Quantiles agree with the previous search to 2.4e-10, at up to 39% more time off the per-gene path. Unreachable branches for infinite limits are gone, and the docstrings describe the steering of rare overlaps as it is. New tests compare limits per coordinate, tails and small boxes, with Dunnett's integral; check that quantile errors are honest over 100 seeds and meet their target; drop infinite limits from a general correlation; and cover a coordinate fixed by the conditioning one, draws that underflow and rays parallel to a face. Each catches a deliberate bug that the earlier tests missed.
…at mislead Testing a fit's contrasts meant multiplying its posterior variance by an unscaled covariance from its internals. After limma's approximate standard errors there was no such covariance, and the fallback, the design's covariance without weights, gave standard errors about three times too large without an error. Counts as Williams sizes named groups "1", "2", ..., a set of groups gave a control by hash order, "(Intercept)" made contrasts against an intercept, and asymmetric or negative covariances gave messages about arguments never passed, or NaN rows. multiple_contrast_test takes a ModeratedFit in place of the estimates, with its covariance, df_total and names, and refuses approximate standard errors. The builders refuse sets and "(Intercept)"; williams_contrasts takes named sizes only. Covariances must be symmetric, a negative contrast variance names its feature, and contrast_matrix words its messages for the estimates through target and remedy, which replaces a second copy of its name matching. The result keeps each feature's correlation of contrasts for simultaneous intervals, and the gene-level table carries the chosen contrast's estimate and standard error. Tests check the fit path against contrasts_fit, the correlation by hand and the intervals against scipy's dunnett, the one-sided gene-level table, a zero-variance contrast among others, and each refusal; each catches a deliberate bug.
…n level The chapter said Bonferroni's p-values were 20% larger than the single-step ones at 5 df, which is how much smaller the single-step ones are, and claimed that 2 to 4 replicates give small moderated df. The guide named a drug wherever its p-value was below 0.05, which keeps no error rate across the genes called, and the chapter read the largest Williams-type t as the number of doses that changed. The chapter gives Bonferroni's excess by Dunnett's integral at 5 to 40 df and for z-tests, points to df_total, runs its example on a simulated dose series with a moderated fit, confirms contrasts at alpha R / G after the screen of the genes, as stageR does (Van den Berge et al. 2017), says what a Williams-type contrast means and that genes, whose direction is unknown, take two-sided tests, and shows simultaneous intervals from the stored correlation. The guide names drugs at 0.0025 and splits its false calls: 27 of 483 genes, 26 of them changed by neither drug. Williams (1971) is cited, and the glossary keeps its order.
No reference checked limits that differ between coordinates, and none checked p-values below 1e-3 for correlations without a product structure, such as Williams-type contrasts of four or more doses, which are the headline one-sided test and the input of the dose-response tests to come. The script adds 96 boxes of product correlations with a limit per coordinate, against Dunnett's integral and in both calibration pools, and 54 complements from 1e-4 to 1e-10 of Williams-type, Tukey's and random contrasts against ALOE (Owen, Maximov and Chertkov 2019), plain Monte Carlo from the mixture of the distributions given each exceedance, written from the paper in the reference process. Both take their inputs from generators of their own, so the other rows keep theirs.
… work With a moderated fit, contrasts that did not match its coefficients asked for estimate_names, which the fit path refuses; and the refusal of "(Intercept)" among the groups advised testing a fit with an intercept as it is, which puts the intercept into each gene's family of contrasts. A fit's mismatch now speaks of the fit's coefficients and points to their names, and the intercept's refusal says to test the other coefficients without it, or to fit a design without an intercept. The chapter says to name the doses' coefficients as the fit's contrasts, as the bulk RNA-seq guide does.
The validation script gained rows of limits per coordinate and of ALOE, and the code it checks changed in this round, so its section, its digest and the totals of the Validation page were out of date. The section opens with the second run of the script on the final code: 118 checks, 106 pass, none failed, with the new rows and calibration pools, while every other row came out as the build's run printed it; the references are listed as bullets, the reference programs' details moved into the script, and the benchmark of the same container gives the timings. The digest of the code that run checked is recorded, and the summary row says which correlations reach which tails.
The guide's saved outputs named a version built from a working tree with changes not yet committed. They now come from the committed code, and print the same results.
The quadrature of boxes of two coordinates uses the nodes and weights of QUADPACK's 15-point Gauss-Kronrod rule, which only a code comment named, while CytePool cites coefficients taken from books in the References section of the module that uses them. The module now cites Piessens et al. (1983), checked against Crossref, OpenAlex and Open Library, and the bibliography holds the book. The validation script's description of its calibrated tier now names the standard errors in its formula, which read as if the values themselves were combined.
The dose chapter says that each dose against the vehicle with control over a gene's several doses, as Dunnett's test gives it, is not implemented. multiple_contrast_test now gives it: the moderated fit of a cell type from dose_response holds the covariance of its dose contrasts, and the single-step test of those contrasts is Dunnett's test. The chapter now tells readers to pass the cell type's moderated fit and its dose contrasts to multiple_contrast_test, with a link to the section that explains the test.
The bulk RNA-seq guide printed the table of the multiple contrast test with 60 floats at full precision. CI checks a guide's saved outputs by comparing printed text exactly, and on GitHub's runners with AVX-512, numpy computes float64 logarithms, powers and exponentials with other last bits than on an AVX2 machine, so those digits can differ there, while they tell the reader nothing. The cell now prints the five genes of smallest p-value, with the contrast, the estimate to 2 decimals and the p-value and adjusted p-value to 2 significant digits, then the error of the p-values as a share of each p-value, at the median and at most. The paragraph after it names the columns that the table holds and the cell leaves out. Each printed value lies at least 2e-4 (relative) from a rounding boundary, and the p-values of the top six genes lie at least 6.7% apart, so differences in the last bits cannot change the text, and the reader sees the digits that carry information.
The dose response guide showed the first five rows of the hepatocytes' spline table as a raw table, with 20 F-statistics and p-values at full precision. CI checks a guide's saved outputs by comparing printed text exactly, and on GitHub's runners with AVX-512, numpy computes float64 logarithms, powers and exponentials with other last bits than on an AVX2 machine, so those digits can differ there, while they tell the reader nothing. The cell now prints the same five genes under the table's column names, which the paragraph after it explains, with the F-statistic to 1 decimal and the p-value and both adjusted p-values to 2 significant digits. Each printed value lies at least 5e-5 (relative) from a rounding boundary, and the p-values of the top six genes lie at least 0.46% apart, so differences in the last bits cannot change the text.
parashardhapola
deleted the
dose/04-multivariate-t-and-contrast-tests
branch
November 20, 2026 13:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A dose series asks, for each gene, which doses differ from the vehicle, with one error rate for the gene's comparisons. In a survey of 130 published single-cell studies with three or more dose levels, 76 analysed dose as separate contrasts against a control, and 70 profiled exactly three dose levels, usually a control and two doses. Clinical dose finding asks the same question: of 96 phase 2 dose-ranging trials in NEJM, The Lancet and JAMA (2018 to 2026) whose primary analysis could be determined, 39 used pairwise tests with multiplicity control, 27 unadjusted pairwise tests and 15 MCP-Mod.
Dunnett's test (each dose against the control; Dunnett 1955), the Williams-type test (the vehicle against the size-weighted mean of the highest one, two or more doses; Bretz 2006) and the MCP step of MCP-Mod refer the largest of correlated t-statistics to the multivariate t distribution (Hothorn, Bretz and Westfall 2008), which CytePool could not compute. Bonferroni's correction ignores the correlation that the shared vehicle creates, 0.5 between two doses of groups the size of the vehicle's: at single-step p-values from 1e-2 to 1e-6, Bonferroni's p-values are 22% to 26% larger than the single-step ones for three such contrasts at 5 degrees of freedom, and 65% to 73% larger for eight. The gap falls as the degrees of freedom grow and the p-values shrink, to at most 7% and 15% for z-tests (Dunnett's integral). An F-test of the contrasts, which PR 2 added, says whether any dose differs, not which.
The p-values also have to be accurate where calls are made. At a false discovery rate of 5%, the Benjamini-Hochberg cut-off for the top gene of 20,000 is 2.5e-6, while mvtnorm, which multcomp and DoseFinding use, defaults to an absolute error of 1e-3 (
GenzBretz(maxpts = 25000, abseps = 0.001); Genz and Bretz 2002). Asked for an absolute error of 1e-12 on three correlated t-statistics with 10 degrees of freedom, mvtnorm 1.4-2 returned 4.03e-6 for a true 4.37e-6, 8% low and twice its own stated error, and 0 for a true 5.9e-9, with a stated error of 0; in a probe with another seed it returned 2.58e-6, 41% low and 16 times its stated error.The solution
Two modules.
cytepool.stats.multiple_contrastsbuilds Dunnett's and Williams-type contrasts and runs single-step multiple contrast tests over every gene, from a moderated fit, or from estimates and their covariances as multcomp'sglhttakes them withparm():df_total; a fit with limma's approximate standard errors of contrasts holds no covariance of its contrasts and is refused. Degrees of freedom need not be integers, as moderated ones are not.table()gives each gene's test as a whole: its most extreme contrast, that contrast's estimate and p-value, and Benjamini and Hochberg's adjustment over the genes.table(contrast)gives that contrast's p-value adjusted for the gene's contrasts, which the documentation reads at stageR's confirmation level, alpha R / G, after the screen of the genes (Van den Berge et al. 2017). Within a gene, p-values never decrease as t becomes less extreme. The result keeps the correlation of each gene's contrasts, from whichmultivariate_t_quantilegives simultaneous confidence intervals; for group means they agree with scipy'sdunnettwithin scipy's own Monte Carlo error.dunnett_contrasts(groups, control=...)names each contrast "dose - vehicle", as multcomp does.williams_contrasts(sizes)takes the size of each named group, weighs the j highest doses by their sizes and names each contrast by the doses it pools, such as "mid to high - vehicle", where multcomp writes "C 2"; tested two-sided for genes, whose direction is not known, Williams-type contrasts test rises and falls at one error rate. Both builders return aContrasts, matched to the estimates by name, and both refuse a model's intercept among the groups, and sets, which have no order.cytepool.stats.multivariate_tgivesmultivariate_t_cdf, the probability of a box and its complement, andmultivariate_t_quantile, the critical value of a max-t test, each with its standard error. The complement is a sum over the coordinates of the chance that each is the first to exceed its limit; each term is computed by conditioning on its exceedance, with randomly shifted lattice points (Genz 1992; Genz and Bretz 2002), which keeps the relative error bounded however small the p-value (Owen, Maximov and Chertkov 2019). Each p-value carries a standard error of at most 1% of itself by default (relative_error). Two contrasts, as a vehicle and two doses give, are computed by a deterministic quadrature to 1e-12 relative, without random numbers."Multiple testing" gains a section, "Several contrasts of one gene", with this example, the Bonferroni gaps by degrees of freedom and simultaneous intervals; "Choosing a method" and the glossary gain entries. The bulk RNA-seq guide tests both drugs against the control of its simulated experiment at one error rate per gene: the max-t test calls 483 genes, 5.4% of them unchanged, against 508 and 4.5% for the F-test, and stageR's rule then names a drug in each called gene, with 27 of the 483 genes (5.6%) carrying a false call, 26 of them genes that neither drug changed. The dose response guide, from PR 7, now prints the first five rows of its spline table rounded, F to 1 decimal and p-values to 2 significant digits, as the bulk guide prints its strongest genes: CI compares printed text exactly, and full-precision floats can differ in their last digits on runners with AVX-512.
Why it matters
PR 7 fits one moderated model of the dose groups in each cell type. PR 8 runs Dunnett's, Williams-type and MCP-Mod tests of that fit through
multiple_contrast_test, takes MCTtest's critical value frommultivariate_t_quantile, finds the lowest effective dose by a closed test, adjusts per-dose contrasts stage-wise after the gene-level screen, and addsoptimal_contraststocytepool.stats.multiple_contrasts. Because each p-value keeps its relative accuracy down to the smallest doubles, the calls that Benjamini and Hochberg make across 20,000 genes change with the seed only for p-values within about 2% of their cut-off.On 20,000 genes and 2 threads, in five runs on three kinds of Modal host, Dunnett's test of 8 doses took 0.60 to 0.88 ms per gene, 76 to 106 times the baseline (the weighted fit of 20,000 genes by 200 samples that
linear_models.py --benchmarktimes), within the target of 1 ms per gene, about 120 baselines. Two contrasts took 0.02 to 0.04 ms per gene. The Williams-type test of 8 one-sided contrasts took 1.13 to 1.67 ms per gene (143 to 193 baselines), above the target on every host. In a probe on 2,000 of these genes, a Williams-type problem took 307 points at the mean against 223 for Dunnett's, and p-values of at least 0.5 took 70% of the Williams-type points (94% of Dunnett's), as the stopping rule holds the smaller of p and 1 - p to its relative error. A tenfold smallerrelative_errortook 12.0 to 12.6 times as long.Validation
validation/multiple_contrast_tests.pyran on Modal twice in one container (4 CPU, 8 GiB, an Intel Granite Rapids host, with the settings that make numpy and OpenBLAS compute as without AVX-512) and printed the same 118 checks both times: 106 pass, 6 information, 3 deliberate differences, 3 controls, none failed. The second run took 7 min 45 s with a peak memory of 475 MiB, as the Validation page reports. Every Monte Carlo problem has its own seed, and CytePool runs atrelative_error=1e-3in the calibrated checks, whose tolerance is 5.5 combined standard errors: the 99.994% point of the t distribution on the 15 degrees of freedom of CytePool's errors, whose two-sided tail is that of 4 normal standard errors. mvtnorm's and multcomp's standard error is their error attribute over 3, as 200 repeated calls measured; scipy's comes from the spread of 16 calls and ALOE's from its own draws.relative_error1e-3 and 369 at 0.01 with fresh seedspmvtmultivariate_t.cdfpmvtat CytePool's quantile;multivariate_t_cdfdunnett_contrasts(23 cases, names too) andwilliams_contrasts(14 cases), 2 to 9 groupscontrMatmultiple_contrast_test: estimates, standard errors and t of 160 features, 4 families, every alternative, df 5 to 40 and the normalglht(parm(b, V, df))multiple_contrast_test: 675 single-step p-values from 1e-3 to 1, of 760summary(glht)adjusted("bonferroni")Information rows: asked for an absolute error of 1e-12 with 1e7 points, mvtnorm's
pmvtreturned 4.025e-6 for a true 4.367e-6 (8% low, twice its stated error of 1.7e-7) and 0 for a true 5.9e-9;qmvt, whose root search keepspmvt's default absolute error of 1e-3, differs from CytePool's 40 quantiles by at most 5.5e-3 relative; scipy's complements for the singular correlation of Tukey's contrasts of 4 groups reach 11.75 times Bonferroni's upper bound, which no probability can exceed, where CytePool agrees with mvtnorm; below 1e-3, which its default absolute error cannot resolve, multcomp's 85 p-values ranged from 216 times smaller to 4 times larger than CytePool's, and were 0 for 10 of them. The largest calibrated difference, 5.48 standard errors at a p-value of 0.0115 of random contrasts with a nearly singular correlation, is multcomp's: a probe on Modal found its value at the script's seed, 0.0097, 6.1 of its own standard deviations below the mean of 30 calls with other seeds (0.0114), which CytePool's 0.01146 matches (0.01147 atrelative_error1e-5). Deliberate differences: non-integer df, whichpmvtrefuses; Williams-type names that say which doses are pooled; Williams with two groups, whichcontrMatrefuses.Modal runs, each an ephemeral app in the image that holds the
r-doseandpythonenvironments, 16 in all and about 3.6 container-hours:contrMat,pmvtwith non-integer, zero and infinite df, the meaning of its error attribute, its tail,qmvtandglht(parm())contrMatwith two groups, non-integer df); the third gave the numbers the design used--benchmark, in one containersummary(glht)stopped on a feature with a very large t; the seeded effects were moderated and each multcomp call guarded--write-fixtures--write-fixtures, and--benchmarkmyst build --strictcheck-outputsof the bulk RNA-seq and dose response guidescheck-outputsof the bulk RNA-seq and dose response guides, and the same check of both guides before their tables were roundeddocs/changed_since_run.pylists 34 other scripts whose checked code changed after their recorded runs. This PR rewords the error messages of two shared helpers,contrast_matrix(which now names "the estimates" where it named the fit's coefficients) andcoefficient_position, which 32 scripts reach; the wording cannot change their results, but the recorded digest counts the strings the code uses, so the digests of all 32 change. Three of them, dose_response_calibration.py, dose_response_tests.py and plotting.py, were current on master; the other 29 were listed on master already, because master changed their code after their runs, as were multiple_testing.py and simulate_dose_series.py, which this PR does not reach. The 34: activity.py, combined_p_values_and_replication.py, composition_tests.py, cube_collection.py, deseq2_tests.py, design.py, differential_expression.py, differential_expression_options.py, dose_axes.py, dose_design.py, dose_response_calibration.py, dose_response_tests.py, dream.py, empirical_bayes.py, gene_set_files_and_ora.py, gene_set_tests.py, gsea.py, interferon_stimulated_pbmcs.py, kenward_roger_and_f_tests.py, linear_models.py, meta_analysis.py, mixed_models.py, multiple_testing.py, negative_binomial_fits.py, pathway_scores.py, pca_and_dot_plots.py, plotting.py, quasi_likelihood.py, rotation_gene_set_tests.py, sample_quality_weights.py, simulate_dose_series.py, sumrank.py, variance_partitioning.py and voom.py.Tests
tests/test_multivariate_t.pyandtests/test_multiple_contrasts.pyhold 131 tests, which run in 17 s once numba has cached the compiled routines; the eight fixture files oftests/data/multiple_contrast_tests/(74 kB) come from the validation script's Modal run and hold only the references' values.contrMat, and Williams' weights by hand;multiple_contrast_testagainst multcomp's estimates, standard errors and t (1e-10) and p-values (5.5 combined standard errors, where multcomp resolves them), and one contrast against limma's moderated t-test (1e-12); a moderated fit's contrasts, weighted and not, againstcontrasts_fitandmoderated_statistics; and the correlation of Dunnett's contrasts against lambda_i lambda_j by hand, with simultaneous intervals and p-values against scipy'sdunnett."less"at t equals"greater"at -t exactly; p-values never decrease as t becomes less extreme; a two-sided limit of 0 gives a complement of exactly 1.tests/test_masked_inputs.py); named contrasts matched by name, pandas included; the one-sided gene-level table; and the examples of the docstrings.Each of 37 deliberate bugs ran in a copy of the package and made at least one test fail: 22 against the two test files as first written, 14 against the tests added in code review, and, after review restructured the code, seven of the first 22 again plus one more against
tests/test_multivariate_t.py. At most three catching tests are listed:test_one_coordinate_is_the_t_distribution(3 cases)test_two_coordinates_match_mpmath,test_two_coordinate_quantiles_match_mpmath,test_two_coordinates_of_correlation_one_are_one_coordinatetest_product_correlations_match_dunnetts_integral_into_the_tail,test_rank_two_correlations_match_the_average_over_directions,test_stated_errors_are_honesttest_errors_stay_honest_deep_in_the_normal_tailtest_errors_stay_honest_with_nearly_collinear_coordinates,test_general_correlations_match_mvtnorm,test_rank_two_correlations_match_the_average_over_directionstest_stated_errors_are_honesttest_equicorrelated_normal_matches_dunnetts_integral,test_product_correlations_match_dunnetts_integral_into_the_tail,test_stated_errors_are_honesttest_errors_stay_honest_with_nearly_collinear_coordinatestest_errors_stay_honest_with_nearly_collinear_coordinatestest_multiple_contrast_test_matches_multcomp,test_product_correlations_match_dunnetts_integral_into_the_tail,test_small_boxes_keep_their_relative_accuracytest_errors_stay_honest_with_nearly_collinear_coordinates,test_product_correlations_match_dunnetts_integral_into_the_tail,test_small_boxes_keep_their_relative_accuracytest_two_contrasts_need_no_random_numbers,test_two_coordinates_match_mpmathtest_two_coordinates_match_mpmathtest_product_correlation_quantiles_match_dunnetts_integraltest_contrast_matrices_match_multcomp,test_williams_weights_by_handtest_contrast_matrices_match_multcomp,test_dunnett_contrasts_take_groups_and_a_controltest_multiple_contrast_test_matches_multcomp,test_the_examples_run"less"tested as"greater"test_less_at_t_is_greater_at_minus_t,test_multiple_contrast_test_matches_multcomptest_a_less_extreme_contrast_never_gets_a_smaller_p_value, added, catches ittest_the_tablestest_multiple_contrast_test_matches_multcomp,test_one_contrast_is_the_moderated_t_test,test_p_values_never_decrease_as_t_becomes_less_extremetest_distinct_limits_match_dunnetts_integral(3 of 4 cases)test_distinct_limits_match_dunnetts_integral(4 cases)test_quantile_errors_are_honest_and_meet_the_targetrelative_errortest_quantile_errors_are_honest_and_meet_the_targetcorrelation[0, 1]; three reading the leading blocktest_infinite_limits_drop_their_coordinates_from_any_correlation(each)test_the_overall_table_of_a_one_sided_testtest_a_contrast_of_zero_variance_makes_its_feature_missingtest_a_negative_variance_is_refused;test_asymmetric_covariances_are_refusedtest_a_moderated_fit_is_tested_with_its_own_covariances(weighted case; both cases)test_the_correlation_of_dunnett_contrasts_and_simultaneous_intervalstest_contrast_builders_refuse(2 cases)test_bounds_hold,test_a_coordinate_fixed_by_the_conditioning_oneandtest_distinct_limits_match_dunnetts_integralCoverage of the final code, with
NUMBA_DISABLE_JIT=1(coverage 7.13.1, branches):cytepool/stats/multiple_contrasts.py100% of 247 statements and 76 branches;cytepool/stats/multivariate_t.py99% of 807 statements and 336 branches. Left: a guard that ends the two-coordinate quadrature when an interval cannot be split, which no input reached; the 200-step cap of the quantile's bisection, reached only by a root at exactly 0 with an error of exactly 0; and the two-sided limit of 0 of the box's slope, which the tests reach compiled. Uncompiled,t_sf_scalarofdistributionsraises at 0 (math.log(0)), which fails four tests that pass compiled, so the coverage run leaves them out, with the five that run thousands of uncompiled Monte Carlo problems.Decisions
relative_error(1%) of the smaller of the probability and the complementcytepool.stats.multivariate_tcytepool.stats.distributionsdistributions.py, whose change would mark them all as changed since their runestimatesandestimate_names, contrasts matched by name; builders that returnContrastsmeans, as first planned; contrasts read by position; plain matrices ascontrMatreturnsparm()does; CytePool never matches named weights by position; names travel with the weights into tablescontrast_standard_errors="limma"(no per-gene covariance), and its fallback, the design's covariance without weights, gave standard errors about three times too large at the median, without an error; the fit path refuses such fits and cannot take an unscaled covariancewilliams_contraststakes a mapping of group name to size onlycontrMat(n)takesestimate_names=["1", "2", ...];dunnett_contrastsalready took namesconfint; no intervalsmultivariate_t_quantile, which agree with scipy'sdunnett, and PR 8's MCTtest critical value needs the same correlationdfwithout a default, not necessarily an integerTrimmed, and where each went:
pmvtwithabseps = 1e-12:pmvtcannot reach them, as the information rows show, so mpmath, Dunnett's integral and ALOE give the tail references, down to complements of 1e-12.confint) and step-down adjustments (Shaffer, Westfall): out; PR 8's lowest effective dose uses its own closed test, and the chapter gives intervals in three lines from the stored correlation andmultivariate_t_quantile.Left for later: resolving p-values near 1 with fewer points while keeping their errors honest, which would bring the Williams-type test closer to Dunnett's cost; and dropping the x^2 map that steers draws of rare overlaps, which a broad check found neither helps nor harms (a pooled root-mean-square z of 1.146 both ways over 183 problems and 30 seeds each) but whose removal needs another validation run.