Skip to content

Decide which genes respond with a gate, and check it against noise - #84

Merged
parashardhapola merged 12 commits into
masterfrom
dose/09-responsive-gene-gate
Nov 21, 2026
Merged

parashardhapola merged 12 commits into
masterfrom
dose/09-responsive-gene-gate

Conversation

@parashardhapola

Copy link
Copy Markdown
Member

The problem

A dose series runs several tests of each gene (#81, #83), and nothing said which of them decides that a gene responds, at which false discovery rate, or how a fold change enters that decision; lowest_effective_doses took a screen and a level of its own, a second name for the same call. Toxicology's workflows decide with a prefilter and cut the estimated fold change after the test, which controls no error rate for the size of a change, and two studies measured what that costs. With zebrafish embryos exposed to the vehicle alone and placed at random in 4 mock concentration groups of 6, BMDExpress's default prefilter, a Williams trend test with a 1.5-fold cut, gave a transcriptomic point of departure, the benchmark dose of the 25th most sensitive gene, in 93% of runs (Lee et al. 2024). With the concentration labels of a real series permuted 100 times, a workflow whose prefilter took an unadjusted Williams p-value found responsive genes in every permutation, where one with a false-discovery-rate-adjusted Williams test found none in nearly all (Bruns et al. 2026). #81's moderated tests control the false discovery rate within a cell type, but at 2 replicates per dose a cell type of 1,000 null genes still had a spline call in 10.2% to 15.5% of datasets, and nothing tested, at a user's own design, whether the genes their analysis calls could come from noise.

The solution

gate= in dose_response, one recorded test of any response at a stated FDR after the dose-blind expression filter, and permutation_check, which reruns the analysis on permuted dose labels:

import cytepool as cp

design = cp.Design(cube.samples, "~ 1")
result = cp.dose_response(cube, design, dose=axis)  # gate: the spline test at 5% FDR
result.table("hepatocyte", "gate")  # p_value, adjusted_p_value, passes_gate
large = cp.dose_response(cube, design, dose=axis, gate={"fold_change": 1.5})
check = cp.permutation_check(result, n_permutations=100, rng=1)
print(check)
PermutationCheck of the gate (spline test, Benjamini-Hochberg within each cell
    type, at 5% FDR): 3 cell types, 2 checked; seed 1
  hepatocyte: 88 genes pass the gate; over 100 random runs of 1.1e21
              assignments, the gate passed genes in 3 runs (0.03), 0.04 genes
              per run; p = 0.0099, the smallest these runs can give
  ...
  • gate= takes {"test": ..., "fdr": ...} with the spline (the default; the dose groups where fewer than two doses are on the axis), the dose groups, Dunnett's, the Williams-type or the MCP test, or {"fold_change": 1.5}: each dose against the vehicle by TREAT (McCarthy and Smyth 2009), so that the threshold is part of each dose's null hypothesis, the smallest of a gene's p-values times its doses (Bonferroni, valid whatever the doses' correlation), then Benjamini-Hochberg across the cell type's genes, recorded as the test "each_dose". A gene passes at an adjusted p-value at most the FDR. The gate's test runs whether tests names it or not and is never left out as a repeat; "linear_trend" and "nonlinearity" are refused, as they test a line or a departure from one. A fold change under dream is refused, as TREAT takes limma's moderated t-tests.
  • What the result keeps. Each tested cell type gains table(g, "gate"), the summary n_genes_passing_gate (NaN where the gate did not run), DoseResponseResult.gate, which dose_response(..., gate=result.gate) takes back, provenance["gate_adjustment"], and the result's cube and design, which the check reruns. The printout names the gate with its adjustments, as the OECD Omics Reporting Framework asks a report to name a prefilter's test, threshold, multiple-testing method and fold change (OECD 2023, Sec. 4.3.4), and counts calls at most the FDR, as the gate does.
  • lowest_effective_doses(group) reads the gate. This PR replaces Test dose trends and candidate shapes, and find lowest effective doses #83's screen= and alpha= with the gate, which no release carried: another test or level is another gate, and the call column is passes_gate, one name for one call. Under a fold-change gate each dose is called by its TREAT p-value, so that the lowest effective dose is the lowest dose whose change beyond the threshold is called.
  • cp.permutation_check and cp.PermutationCheck. In each cell type, the dose labels of the samples it uses are permuted among the samples that the design treats alike: samples in blocks of the same row of the design's other terms (so "~ sex" permutes within each sex), the units of one random intercept whole when each holds one dose, in blocks of units with the same rows, and the doses within units otherwise, so that every run keeps the dose groups' sizes, the knots and the filter's genes. dose_response runs again with the result's options and gate, fitting the gate's test alone with the result's seed, and the summary gives the genes the true doses pass, the share of runs that pass genes, the genes per run, the permutation p-value (1 + b) / (1 + B) (Phipson and Smyth 2010), which answers whether the true doses pass more genes than permuted ones, and smallest_possible_p_value. A design with at most n_permutations other assignments runs each of them, which makes p exact. Runs whose gate gives the true doses' p-values up to rounding tie, and the smallest possible p-value is (1 + ties) / (1 + runs): at the vehicle and two doses in duplicate, the 6 orders of the three groups' labels tie under the dose-group gate and the default spline, so p cannot fall below 6/90 = 0.067; at one treated dose every gate ties the swap of the dose and the vehicle; and the spline ties the reflection of doses that sit symmetrically on the axis. Before the runs, the true doses run once through the same path and must give the result's gate, the same genes, p-values within a relative 1e-9 and the same calls; each run's assignment is recorded, so any run can be made again.
  • Shared code. The options of an analysis read back from its record come from cytepool/_reruns.py, which sumrank now uses too, and the recorded seed's rule moves to _options.recorded_seed; the check reuses dose_response's analysis map, its wording of the gate and its result's reasons, and the cube's checks of groups.

The Dose response chapter gains "Which genes count as responding" and "Could the calls come from noise?", and its lowest-dose section, defaults, limits, checks and further reading follow the gate; Choosing a method gains a question and switches; the glossary gains "Gate" and "Permutation check"; the bibliography Lee et al. (2024) and Bruns et al. (2026); the API reference a page for the check. The guide "Dose response per cell type" gains two sections: on its hepatocytes the 1.5-fold gate passes 75 of the 215 genes whose true change exceeds 1.5-fold and no other gene, where the default gate with a 1.5-fold cut on the estimated changes keeps 216 genes, 38 of them never beyond 1.5-fold in truth; and the check finds no run that passes as many genes as the true doses in any cell type, so each p-value is 1/101, with 47 of the 51 genes that passed in the hepatocytes' runs responsive in truth, which is why the guide and the chapter read the p-value, not the share of runs, as the answer.

Why it matters

One recorded decision now says which genes respond, with its test, level, fold change and adjustments, and later stages read it. permutation_check answers at the user's own design the question that Lee et al. and Bruns et al. asked of whole workflows: could the genes that the gate passes come from noise?

The gate builds on #81's tests and expression filter, #83's multiple contrast tests and stage-wise calls, cytepool.stats.empirical_bayes.treat, already validated against limma, and #79's simulator, which the validation and the guide use. PRs 10 and 11 can put the gate's call beside each curve and benchmark dose, whichever of them merges after this PR; PR 15's gene-set roll-ups and points of departure will admit only the genes the gate passes, and PR 15 adds benchmark doses and points of departure to the check's runs, a stage PR 11 left to it; PR 17 weighs the default gate on the full grid of designs; and PR 23 reruns the gate on resampled data.

Validation

Each script ran twice in one Modal container (4 CPU, 8 GiB) and printed the same rows both times. validation/dose_gate.py (new) printed 26 checks, 26 pass; the recorded run, at the final code on an AMD Zen 3 host as the Validation page's, took 1 min 45 s with a peak memory of 737 MiB. R sees the counts, cells, doses, the design's variables and the settings, and nothing CytePool computed; it fits limma 3.62.1 and edgeR 4.4.0 from the counts, stageR 1.28.0 calls the lowest effective doses, R builds the exchangeable units from the sample table itself, counts and enumerates the assignments, and refits limma on every assignment of two designs, counting the ties among its refits; variancePartition 1.36.2 fits the dream designs. The spline gate's p-values are #81's F-tests, which dose_response_tests.py checks against limma, so R checks that gate's calls on CytePool's p-values.

Check family Reference Largest difference Tolerance Outcome
The dose-group gate's p-values, exact and limma's standard errors, with and without robust priors, three series, limma-voom and limma-trend limma from the counts: lm.wfit under voom's weights or classifyTestsF, pf on each gene's df 6.6e-13; robust 1.3e-9 1e-10; 1e-6 4 pass
The genes each gate passes at 5% and 10% FDR (16,504 calls), and the genes tested p.adjust(p, "BH") <= fdr on R's p-values, and on CytePool's for the spline identical identical 3 pass
TREAT p-values of each dose at 1.2- and 1.5-fold limma treat(fit, lfc = log2(fc)), topTreat 3.2e-13; robust 9.7e-10 1e-10; 1e-6 3 pass
The fold-change gate's p-values at 1.2- and 1.5-fold pmin(1, m * apply(P, 1, min)) on treat's p-values 6.0e-13; robust 1.0e-9 1e-10; 1e-6 3 pass
The genes the 1.5-fold gate passes and their lowest effective doses (2,859 genes); the stage-wise adjusted p-values there p.adjust; stageR stageWiseAdjustment(method = "holm") on R's TREAT p-values identical; 1.1e-9 identical; 1e-6 3 pass
The units permuted and the number of assignments, every series and cell type, up to 1.4e11 R's reading of the sample table and the design; products of choose() identical identical 2 pass
Every assignment of the duplicate series (90) and of the sexes (36), each run once, two gates each; the genes each run passes; the share, the p-value and the smallest possible p-value from the ties (6/90 and 2/90; 2/36 under both gates) R's enumeration of each block's label vectors; limma refitted by R on each assignment identical; 0 identical; 1e-10 4 pass
One recorded run of each limma series: its gate p-values, the genes passed, the check's count against dose_response on the permuted design limma refitted by R on the recorded doses 5.3e-14; identical 1e-10; identical 3 pass
dream on donors (doses within units) and animals (whole units in blocks by sex): the dose-group F-statistics of the true doses and of a recorded run variancePartition dream, topTable 5.3e-8 1e-6 1 pass

validation/dose_contrasts.py, run again since lowest_effective_doses now reads the gate, printed the same 80 checks as #83, 71 pass, 7 deliberate differences and 2 information, its rows of lowest effective doses now read through the spline and dose-group gates, with the same values: identical calls to stageR's, adjusted p-values within 5.2e-12, and every gene's lowest effective dose identical to the enumerated closure's (6,579 and 6,340 genes). Its recorded run, at the final code, printed the same table row for row; the second run took 3 min 26 s with a peak memory of 940 MiB.

validation/dose_response_calibration.py printed 139 checks, 14 pass, 111 information, 6 known limits and 8 controls, 0 failed, #81's and #83's 101 rows as recorded; the second run took 1 h 2 min 26 s with a peak memory of 451 MiB, on an Intel host with the harness's AVX2 settings, where the earlier runs on AMD Zen 3 took 1 h 57 min. The gates' share of null cell types with a gene passing at 5% FDR, 1,000 datasets of 2 cell types per design cell, against 5% plus 4 binomial standard errors of the 2,000 cell types each share counts, 6.95%:

Design The spline test, the default The dose groups 1.5-fold, by TREAT Outcome
K = 4, 2 replicates 0.155 0.183 0.011 2 known limits, 1 pass
K = 4, 3 replicates 0.069 0.075 0.001 1 known limit, 2 pass
K = 9, 2 replicates 0.102 0.144 0.007 2 known limits, 1 pass
K = 9, 3 replicates 0.068 0.072 0.002 1 known limit, 2 pass

The spline gate holds the bound at 3 replicates and exceeds it at 2, and the dose-group gate exceeds it at both, the tail that #81 measured for the moderated tests; those six rows print the bound and "known limit" without failing the script, and the default gate stays as it is until PR 17 weighs it on the full grid of designs. The bound counts the 2,000 cell types each share counts, whose between-dataset standard error matches the binomial one; a bound from the 1,000 datasets, 7.76%, would lie 5.7 standard errors above 5%. permutation_check of the default gate, 100 runs on each of the first 100 null datasets, passed genes as often as the other 900 null datasets did: a mean share of runs of 0.162, 0.072, 0.105 and 0.064 against 0.154, 0.068, 0.102 and 0.069, each within 4 standard errors of the difference (2 pass per design cell), and its p-value was at most 0.05 in 1.5% to 6.0% of null cell types, within 11.2%. Between series of one design the share varies: at 4 dose groups in duplicate its standard error over 100 series, 0.0076, gives a standard deviation of 0.08 around 0.16. On the power datasets, where a fifth of the genes respond, every check gave a p-value at most 0.05; the 1.5-fold gate found 0.24 to 0.45 of the genes of a 2 log2 change with a false discovery proportion of 0.000 to 0.011, a false call a gene never beyond 1.5-fold. In the vehicle-only design of Lee et al. (2024), 4 mock dose groups of 6 samples, 200 datasets, the default gate passed a gene in 17 of 400 cell types (4.3%), where an unadjusted Williams-type p-value below 0.05 with a 1.5-fold change at a dose passed one in all 400. The four new controls, the unadjusted Williams-type gate, passed a gene in all 400 null cell types of each design cell, above the 9.36% that 400 cell types allow.

validation/dose_gate.py --benchmark, on 2 threads, the median of three checks after a first one, on one cell type of 20,000 genes x 24 profiles (the vehicle and 7 doses x 3), against a baseline, the weighted fit of 20,000 genes x 200 samples (0.207 s in the same run): 100 runs took 70.6 s with the spline gate, 340 baselines, within a target of 350 baselines per 100 runs, where runs that fitted every test of the result took 101.6 s (477); 140.6 s with the 1.5-fold gate (678), which fits each run's cell type once more for TREAT; and 24.1 s per run with the Williams-type gate, as information.

Modal runs, each an ephemeral app through the reference harness (times CET; 17:58 to 22:55 on 20 November during the build, 00:29 to 01:43 on 21 November after code review, and the two documentation jobs of the final checks at 03:22 and 03:26):

Start What Why Resources Duration Result
17:58 validation/dose_gate.py on the prototype (PR 8 at 03d578d plus this PR's code) First run of the R programs 4 CPU, 8 GiB, Intel with AVX-512 57 s Failed: R iterated a data frame of jobs; jobs keyed by name
18:01 The same Debugging the R side 4 CPU, 8 GiB, AMD Zen 3 89 s Failed: stageR's "holm" on one hypothesis (sex, one dose); no stage-wise step there
18:05 The same Debugging 4 CPU, 8 GiB, AMD Zen 3 174 s Failed: R's NA written as strings; na = "null"
18:08 The same Debugging 4 CPU, 8 GiB, AMD Zen 3 174 s 29 checks, 2 failed: R's partition key not canonical; dream F 12% to 17% apart
18:12 The same, with dream diagnostics Find the dream gap 4 CPU, 8 GiB, AMD Zen 3 203 s Smallest p-values identical; dream gap from voom's span (CytePool default vs R's 0.5)
18:16 The same, span 0.5 on both sides Check 4 CPU, 8 GiB, AMD Zen 3 190 s 29 checks, 0 failed
18:17 validation/dose_gate.py --write-fixtures on the prototype Write tests/data/dose_gate/limma.json 4 CPU, 8 GiB, Intel with AVX-512 106 s 29 checks, 0 failed; fixture created
18:50 validation/dose_gate.py --write-fixtures, twice in one container, at d8ff3aa The gate's validation table and its reproducibility 4 CPU, 8 GiB, AMD family 23 model 49, no AVX-512 275 s (runs 168 s and 106 s, peak 740 MiB) 29 checks, 0 failed, both runs identical; fixture unchanged
18:50 validation/dose_contrasts.py --write-fixtures, twice in one container, at d8ff3aa PR 8's validation with the gate's lowest effective doses 4 CPU, 8 GiB, Intel family 6 model 173, AVX-512 323 s (runs 199 s and 124 s, peak 1,024 MiB) 80 checks: 71 pass, 7 deliberate differences, 2 information, 0 failed; tables identical; fixtures unchanged
18:50 validation/dose_response_calibration.py twice in one container, at d8ff3aa The gates' errors under the null, the check's agreement with them, the 1.5-fold gate's power and the vehicle-only design 4 CPU, 8 GiB, AMD family 25 model 1 (Zen 3), no AVX-512 14,164 s (runs 7,145 s and 7,010 s, peak 995 and 451 MiB) 139 checks, 0 failed: 16 pass, 111 information, 4 known limits (the spline and dose-group gates at 2 replicates), 8 controls; both runs the same rows; PR 7's and PR 8's rows as recorded
18:58 validation/dose_gate.py --benchmark The check's run time against a weighted baseline 4 CPU, 8 GiB, AMD family 25 model 1 (Zen 3), no AVX-512 1,954 s (peak 1,221 MiB) spline gate 100.1 s for 100 runs, 1.5-fold 195.7 s, Williams-type 155.5 s for 5 runs; timed against the ordinary least squares fit (0.042 s), not the weighted baseline, so run again
19:10 make check-outputs of the dose guide on an AVX-512 host (try 1) CI runs the guide's outputs on AVX-512 runners 4 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 5 s stopped at once: no AVX-512
19:12 The same (try 2) The same 4 CPU, 8 GiB, Intel family 6 model 173, AVX-512 107 s outputs current
19:30 to 19:32 The guide's check on AVX-512, tries 3 to 8, after the guide gained a cell The same 4 CPU, 8 GiB 4 to 9 s each all on AMD family 25 model 1 hosts without AVX-512; stopped at once
19:30 validation/dose_gate.py --write-fixtures twice in one container, after the printout's wording changed The final code's run (superseded: the benchmark's baseline changed after it) 4 CPU, 8 GiB, Intel family 6 model 143, AVX-512 209 s (runs 125 s and 84 s, peak 738 MiB) 28 checks, 0 failed, both runs identical; the empty row of counts above 2^53 dropped; fixture unchanged
19:34 validation/dose_gate.py --benchmark, against the weighted baseline The check's cost against its target 4 CPU, 8 GiB, AMD family 25 model 1 (Zen 3), no AVX-512 2,052 s (peak 1,252 MiB) weighted baseline 0.213 s (OLS 0.031 s); spline gate 101.6 s for 100 runs, 477 baselines (target 350, blocks above 525); 1.5-fold 209.0 s, 981; Williams-type 164.1 s for 5 runs, 27.35 s per run
19:37 The guide's check on AVX-512, try 9, 8 CPU The same, after the guide gained a cell 8 CPU, 8 GiB, Intel family 6 model 173, AVX-512 110 s outputs current
20:14 validation/dose_gate.py --write-fixtures twice in one container, at the final code The recorded run 4 CPU, 8 GiB, AMD family 25 model 1 (Zen 3), no AVX-512 279 s (runs 168 s and 101 s, peak 740 MiB) 28 checks, 0 failed; both runs and the Intel run at 19:30 printed the same rows; fixture unchanged
20:20 CI's documentation job (Node 24, ruff on docs, make -C docs html, the guide's check-outputs) The docs workflow, which needs Node.js this machine lacks 8 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 17 s Failed at ruff format --check docs: one line of the guide's new cell; formatted, executed again
20:25 CI's documentation job again The docs workflow after the guide's formatting 8 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 71 s ruff clean, Sphinx API reference built; MyST stopped on the two placeholders that await the calibration's numbers, read as citations; nothing else
22:52 CI's documentation job at the final docs The docs workflow 8 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 222 s ruff on docs clean, make -C docs html passed (Sphinx with warnings as errors, MyST strict), the guide's outputs current
22:55 The guide's check on AVX-512, tries 15 to 18, 8 CPU, after the formatting of one line The same 8 CPU, 8 GiB 0 to 4 s each all on AMD family 25 model 1 hosts without AVX-512; stopped at once (the outputs were confirmed on AVX-512 at 19:37, and the formatting changed no output)
00:29 validation/dose_gate.py --write-fixtures, twice in one container, at b33f62f Code review's changes: ties counted from R's refits, the sexes refitted, the animals split by sex, no spline p-value rows 4 CPU, 8 GiB, AMD family 25 model 1 (Zen 3), no AVX-512 282 s (runs 175 s and 105 s, peak 737 MiB) 26 checks, 0 failed, both runs identical; fixture rewritten, 406 KB
00:29 validation/dose_response_calibration.py, twice in one container, at b33f62f The bound from the 2,000 cell types each share counts; the check against the 900 null datasets it did not run on 4 CPU, 8 GiB, Intel family 6 model 173, AVX-512, with the harness's AVX2 settings 7,605 s (runs 3,853 s and 3,746 s, peak 1,002 and 451 MiB) 139 checks, 0 failed: 14 pass, 111 information, 6 known limits, 8 controls; both runs the same rows; 121 rows as the earlier run, and the 20 rows of the new bounds and comparison
00:29 validation/dose_gate.py --benchmark at b33f62f The check's cost with runs that fit the gate's test alone 4 CPU, 8 GiB, AMD family 25 model 1 (Zen 3), no AVX-512 1,589 s (peak 1,208 MiB) weighted baseline 0.207 s; spline gate 70.6 s for 100 runs, 340 baselines (target 350); 1.5-fold 140.6 s, 678; Williams-type 24.13 s per run
00:47 validation/dose_contrasts.py --write-fixtures, twice in one container, at b33f62f Its checked code changed again (the gate's wording and refusals) 4 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 557 s (runs 350 s and 206 s, peak 1,022 MiB) 80 checks: 71 pass, 7 deliberate differences, 2 information, 0 failed; the recorded table row for row; fixtures unchanged
01:38 CI's documentation job at fc66d1c (try 1) The docs workflow after code review's documentation 8 CPU, 8 GiB under a minute Stopped before running: the full test suite, starting here, rebuilt cytepool/_version.py while the worktree uploaded
01:39 CI's documentation job at fc66d1c (try 2), the harness's AVX2 settings unset The same 8 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 208 s (peak 916 MiB) ruff on docs clean, make -C docs html passed (Sphinx with warnings as errors, MyST strict), the guide's outputs current
01:43 The guide's check-outputs on AVX-512, tries 1 and 2, the harness's AVX2 settings unset CI runs the guide's outputs on AVX-512 runners 8 CPU, 8 GiB 4 s; 96 s try 1 on an AMD host without AVX-512, stopped at once; try 2 on Intel family 6 model 173 with AVX-512: outputs current
03:22 CI's documentation job at 8686bb3, the harness's AVX2 settings unset The final checks: make -C docs html needs Node.js, which this machine lacks 8 CPU, 8 GiB, Intel family 6 model 106, AVX-512 173 s (peak 913 MiB) ruff on docs clean, make -C docs html passed, the guide's outputs current on AVX-512
03:26 The same at 77c380b, which rewords the benchmark's paragraph on the Validation page The final head 8 CPU, 8 GiB, AMD family 25 model 1, no AVX-512 240 s (peak 917 MiB) ruff on docs clean, make -C docs html passed, the guide's outputs current

docs/changed_since_run.py lists 39 scripts on this branch: 34 of the 35 that master lists, all but the calibration, which this PR records, and anndata_conversion.py, distributions.py, normalization.py, pool_cells.py and smoothing.py, which reach cytepool/_options.py, whose only change is the new recorded_seed. This PR records dose_gate.py, dose_contrasts.py and the calibration, each run at the final code.

Tests

New tests: 28 in tests/test_permutation_checks.py (51 with their cases) and 15 in tests/test_dose_responses.py, with the two new callables in the masked-input and top-level registries and gate_text among the masked-input inventory's unexported callables; #83's tests now read the gate. They check against independent references: limma 3.62.1, edgeR 4.4.0 and stageR 1.28.0 in R, through tests/data/dose_gate/limma.json (406 KB, written by validation/dose_gate.py --write-fixtures): the dose-group gate's p-values and calls under voom and trend, with exact and limma's standard errors, and R's adjustment of the spline gate's p-values; TREAT with robust and trend moderation, the fold-change gate's genes by name and stageR's stage-wise p-values; R's own units, counts and enumerations of assignments, read from the sample table; and the genes that every run of a duplicate design (89 runs) and of a design of two sexes (35) passes, refitted by R, with the ties R finds among its refits. Beside them: brute force over itertools for the assignments of nine block structures (sexes; animals nested in dose, with and without sexes; donors crossed with dose, with and without plates; a numeric covariate; units of unequal size); chi-squared tests of uniform assignments on one block and on two of the same size; ties against renamings, swaps and reflections worked out by hand (one treated dose under three gates, 2/6; Dunnett's test of two doses, 2/90; a spline through four groups of unequal size, 1/180; sexes with unequal counts, 1/9; 0, 5 and 10 on a linear axis, 2/90); counts put into the summary by hand; runs made again by dose_response from their recorded assignments and the result's seed, every run that passes genes under nine option sets, Dunnett's and the MCP gate among them, a Monte Carlo gate and dream; the guard against a forged setting, another cube, p-values moved by 1e-6 and a flipped call, with p-values moved by 1e-12 accepted; the gate worked by hand from a cell type's fit; and counts derived by hand, such as the 42 of 89 assignments that would confound a dose with a lane.

Mutations, each run on its own against the two test files and each caught; 21 ran during the build, and those on code that code review reshaped ran again on the reshaped code:

Mutation Caught by
Calls at < instead of <= test_the_gate_by_hand
The printout counting calls below the FDR test_the_printout_counts_calls_at_the_fdr
Bonferroni over the doses of the design; the largest p-value of the doses; the fold change taken as a log2 threshold; the lowest effective doses on the each-dose t-tests under a fold change; the fold-change gate's genes read gene-major test_the_fold_change_gate_matches_treat
TREAT without the trend; TREAT without robust moderation test_treat_moderates_as_the_fit
Robust moderation dropped from the lowest effective doses under a fold change test_lowest_effective_doses_under_a_robust_fold_change_gate
The p-value without the + 1; runs passing more genes only test_the_summary_by_hand, test_each_run_matches_r_on_every_assignment
The true assignment among the runs; permutations counted instead of assignments test_units_and_assignments_match_r
One block across the strata; nested units permuted as samples; whole units blocked by size alone; doses within units blocked by unit alone test_assignments_by_brute_force
Units of unequal size in one block; samples outside the cell type permuted test_units_of_unequal_size_keep_the_dose_groups
One permutation reused for blocks of one size test_random_runs_are_uniform
Every assignment only when A - 1 < n_permutations test_every_assignment_runs_when_the_runs_allow_it
Ties never counted; ties counted by the genes passed alone; the smallest possible p-value without the true doses test_one_treated_dose_ties_with_its_swap, test_designs_without_renamings_that_tie and four more tie tests
The guard dropped; the guard bit for bit; the guard blind to calls test_a_result_its_doses_do_not_give_again_is_refused (bit for bit also the tie tests)
Runs with fresh seeds, not the result's test_a_monte_carlo_gate_runs_with_the_result_s_seed, added when the mutation survived the first pass, whose option sets had two treated doses, where the multiple contrast tests take no random numbers
Runs fitting the result's tests test_runs_fit_the_gate_s_test_alone
Groups named twice taken once test_refusals
The gate's reasons not the result's test_cell_types_that_cannot_be_checked
The expression filter turned off read back as None test_a_run_is_dose_response_on_the_permuted_doses
The seed from Python's hash of the cell type test_runs_do_not_depend_on_threads_or_other_cell_types
The gate's test left out as a repeat test_the_gate_s_test_runs_when_tests_do_not_name_it
The batch passed to the runs test_runs_leave_the_batch_out

Coverage (coverage 7.13.1, branches) over the tests of dose responses, permutation checks, sumrank, masked inputs and the top level (1,918 tests): every new statement of cytepool/permutation_checks.py (294, all branches), cytepool/dose_responses.py (156) and cytepool/_reruns.py (9) runs. The pass ran with numba's JIT on, since none of the new code is compiled.

Checks at the final head, 77c380b, on master 9fa2cc1 (#83), which has not moved: the full suite 6,111 passed and 18 skipped (zstd needs Python 3.14) in 3 min 18 s; ruff check and ruff format --check on cytepool tests validation docs clean; mypy cytepool clean (96 files); make -C docs check-outputs PAGES="guides/dose_response" found the guide's outputs current here; make -C docs html built the Sphinx API reference here with warnings as errors and stopped at MyST, which needs Node.js, so CI's documentation job (Node 24, ruff on docs, make -C docs html with MyST strict, the guide's check-outputs) ran on Modal and passed, at 77c380b on an AMD host and at 8686bb3, which differs only in two sentences of the Validation page and a docstring, on an Intel host with AVX-512 with the harness's AVX2 settings unset, as on GitHub's runners. docs/changed_since_run.py still finds the three scripts this PR records current.

Decisions

  • One gate per result, at a stated FDR within each cell type, its call in the gate's table and in the lowest effective doses only, so that Test which genes respond to a dose in each cell type #81's and Test dose trends and candidate shapes, and find lowest effective doses #83's tables keep their columns. Benjamini-Hochberg within each cell type, as each test's adjusted_p_value is; adjusted_p_value_across_groups stays in every table.
  • A fold change by TREAT at each dose, with Bonferroni over the doses, rather than a cut on the estimated changes after the test, which controls no error rate for the size of a change. On the chapter's hepatocytes the 1.5-fold gate passes 24 of the 93 genes beyond 1.5-fold and no other gene, where the cut keeps 86, 9 of them never beyond 1.5-fold.
  • The name gate, as PRs 10, 11 and 15 use it; the docstring, the chapter and the glossary say how it differs from NTP's ANOVA gate, a test of the data set, and from BMDExpress's prefilter.
  • lowest_effective_doses reads the gate and loses Test dose trends and candidate shapes, and find lowest effective doses #83's screen= and alpha=, which no release carried: stage-wise values hold only at the gate's level, so another level is another gate. responds becomes passes_gate.
  • The check reruns the public dose_response, with the options read back from the result's record, so that a run is the user's analysis. Each run fits the gate's test alone, with the result's seed, so that the count is a fixed function of the labels and the permutation p-value stays exact under a Monte Carlo gate; each cell type's true doses first run through the same path and must give the result's gate within rounding, the same genes, p-values within a relative 1e-9 and the same calls. A bit-for-bit guard, tried first, refused a result checked on another CPU, whose arithmetic moved the p-values by 1e-13 with the same calls, and forced each run to fit every test of the result.
  • Ties counted from the runs. A run ties when its gate gives the true doses' genes, p-values and calls within that rounding, and the smallest possible p-value is (1 + ties) / (1 + runs). A rule about renamings missed the swap of the vehicle and a single treated dose, under which the fold-change, Dunnett and Williams-type gates test the same difference, the reflection of doses that sit symmetrically on the axis, and renamings drawn among random runs. Dunnett's test of three or more treated doses gives its renamings p-values apart by its Monte Carlo error, so they do not count.
  • Exchangeable units from the design: blocks of the same row of the design's other terms; one random intercept's units whole when each holds one dose, in blocks of units with the same rows, and the doses within units otherwise; more than one random effect or a random slope refused; a run in which the gate does not run refused, never left out. Every assignment runs when there are at most n_permutations others.
  • The p-value is the answer. The docstrings, the chapter, the glossary and the guide read the permutation p-value as the test of whether the true doses pass more genes than permuted ones, and the share of runs that pass genes as how often the gate passes genes with these samples' doses shuffled: it varies between series of one design, and estimates how often noise alone passes genes only where no gene responds.
  • The printout says what the runs count: "the gate passed genes in N runs", since responses scrambled over the permuted groups pass in some runs; the chapter's Limits give the guide's 47 of 51 and the calibration's share on series with responsive genes.
  • The default gate is unchanged at 2 replicates per dose, where its error under the null exceeds the 6.95% that 2,000 cell types allow at 5%: defaults change only on evidence from the full grid of designs, which PR 17 runs. The rows print the bound and "known limit", the Validation page counts them as such, and the chapter points to permutation_check. The bound counts the 2,000 cell types each row's share counts, whose between-dataset standard error equals the binomial one; with it, the dose-group gate's two 3-replicate rows (7.50%, 7.25%) are known limits too, and Choosing a method states that gate's excess where it offers it.
  • The check's cost: 340 baselines for 100 runs with the spline gate, within a target of 350, since each run fits the gate's test alone; the fold-change gate fits each run's cell type once more for TREAT (678), and a multiple contrast gate pays its test in every run (24 s per run with the Williams-type test), both as information.
  • Trimmed: a mode="vehicle" that places vehicle samples in mock dose groups (a vehicle-only series given mock doses in its design runs through dose_response and the check, and the chapter says so); a gate adjusted across cell types; TREAT under dream, for which variancePartition offers no reference to validate against, so a fold change under dream is refused with the reason; benchmark doses and points of departure in the check's runs, which PR 15 adds; and admitting a cell type's genes only when the check's p-value is at most the level, which could admit nothing at the vehicle and two doses in duplicate (smallest possible p-value 6/90) and which PR 17 weighs as a default if the default gate fails there.

A dose series runs several tests of each gene, and nothing said which of them decides that a gene responds, at which false discovery rate, or how a fold change enters that decision. Toxicology's prefilters cut the estimated fold change after the test, which controls no error rate for the size of the change, and lowest_effective_doses took a screen and a level of its own, a second name for the same call.

dose_response takes gate=, one test of any response at a stated FDR after the dose-blind expression filter: the spline by default, the dose groups, Dunnett's, the Williams-type or the MCP test, or, with a fold change, each dose against the vehicle by TREAT with Bonferroni over the doses and Benjamini-Hochberg across genes. The gate's test runs whether tests names it or not and is never left out as a repeat. Each cell type gains the table "gate" with passes_gate, the summary n_genes_passing_gate (NaN where the gate did not run), and the result keeps the gate, its cube and its design; the printout names the gate and its adjustments, and counts calls at most the FDR, as the gate does. lowest_effective_doses reads the gate: under a fold change it calls each dose by its TREAT p-value. The seed rule moves to the options' helpers.

One recorded decision now says which genes respond, reproducibly, and later stages (roll-ups, points of departure, the permutation check) read it.
A Williams prefilter with a 1.5-fold cut gave a point of departure from vehicle-only zebrafish data in 93% of runs, and an unadjusted prefilter found responsive genes in every one of 100 label permutations (Lee et al. 2024; Bruns et al. 2026). With 2 replicates per dose, a moderated gate at 5% FDR still passes genes in 10% to 16% of null cell types, and nothing showed a user how often noise passes the gate at their own design.

permutation_check(result) permutes each cell type's dose labels among the samples the design treats alike (samples in blocks of the same row of the design's terms; units of a random intercept whole when each holds one dose, within units otherwise), reruns dose_response with the result's settings and gate on each labelling, and reports per cell type the share of runs that pass genes, the genes per run and a permutation p-value (Phipson and Smyth 2010). Designs with few assignments run every one, which makes the p-value exact, and the summary gives the smallest p-value such runs can give: 6/90 for an order-free gate at three dose groups in duplicate. Each cell type first reruns its true doses and must reproduce the result's gate p-values bit for bit; each run's assignment is recorded. The options of an analysis read back from its record now come from one helper, which sumrank shares.

Users see at their design whether the genes a gate passes stand out from noise.
The gate's p-values, TREAT at each dose, the lowest effective doses under a fold change and the assignments of dose labels that permutation_check runs needed references independent of CytePool, and the gates' error under the null needed rows in the calibration.

validation/dose_gate.py compares, on series simulated from fixed seeds, each gate's p-values and calls with limma 3.62.1 fitted by R from the counts with its own knots (exact under voom's weights and with limma's standard errors), each dose's TREAT p-values with treat and topTreat, the fold-change gate's p-values and calls, and its lowest effective doses with stageR 1.28.0; the units, counts and enumerations of assignments with R's own reading of the sample table; the genes each of the 89 runs of the vehicle and two doses in duplicate passes with limma refitted by R on every assignment; and dream's F-statistics with variancePartition 1.36.2. The calibration script adds the gates' shares of null cell types with a call (the bound 7.76% a check at 3 replicates and a known limit at 2), the permutation check's agreement with them on 100 null datasets per design cell, the share of its p-values at most 0.05, the unadjusted Williams-type gate as a control, the 1.5-fold gate's power and false discovery proportion, and the vehicle-only design of Lee et al. (2024).
The check's printout said that noise passed genes in its runs, but where genes respond, their responses, scrambled over the permuted dose groups, still pass in some runs: in the dose guide's hepatocytes, 47 of the 51 genes that passed in the runs respond in truth. A coverage pass also found paths without a test: results that named their tests, whose runs must fit the same contrasts; blocks of units that each hold one dose; the printout's lines for a fold-change gate and for runs that cannot reach a small p-value; and the class's refusals. No test showed that the runs leave the batch out.

The printout now says that the gate passed genes in so many runs, and the docstrings say that the share then overstates how often noise alone passes genes. New tests cover each of those paths: the batch test counts by hand the 42 of 89 assignments that would confound a dose with a lane, and the check of a result that named its tests reproduces its gate p-values bit for bit. dose_responses.py and permutation_checks.py are now covered in full.
The check's docstring cited Bourgon, Gentleman and Huber (2010) without the work in its References, the private module of reruns had no reason for citing nothing, and Sphinx read three property docstrings, whose first phrase ended in a colon, as types, which fails the API reference's build.

The References list the work, the documentation tests give the module its reason, and the three docstrings open with a sentence. The check's docstring gives its measured cost: about 1 s per run of 20,000 genes and 24 samples with the spline gate on two threads, twice that with a fold change.
The dose chapter, the guide and the reference pages said nothing of which test decides that a gene responds, how a fold change enters that decision, or how often noise passes it at a series' design, and the lowest effective doses still spoke of a screen.

The chapter gains "Which genes count as responding", with the gate's tests, its table and record, and a fold change by TREAT at each dose against a cut on the estimated changes, and "Could the calls come from noise?", with the check's units, its runs, the smallest p-value of small designs (6/90 at the vehicle and two doses in duplicate), the vehicle-only design of Lee et al. (2024) and the check's cost; its lowest-dose section, defaults, limits, checks and further reading follow the gate. Choosing a method gains a question, the glossary "Gate" and "Permutation check", the bibliography Lee et al. (2024) and Bruns et al. (2026), and the API reference a page for the check. The guide chooses a gate against the truth (the 1.5-fold gate passes 75 of the 215 hepatocyte genes beyond 1.5-fold and no other gene) and checks the calls against permuted doses, remaking the runs that passed genes: 47 of their 51 genes respond in truth. Tests run the chapter's examples and check its numbers against the simulator's truth and the check's records.
The Validation page had no section for the gate and the permutation check, the calibration's section no rows for the gates and the check, and no outcome for the rows at 2 replicates that print their bound and exceed it. The gate's validation compared counts of assignments above 2^53 that none of its series has, and its benchmark timed the ordinary least squares fit rather than the epic's weighted baseline.

validation/dose_gate.py drops the empty row and stops if a count reaches 2^53, and times its benchmark against the weighted fit of 20,000 genes x 200 samples. The page gains the section "The gate and its permutation check" (28 checks, 28 pass), its summary row and its commands; the calibration's 139 checks (16 pass, 111 information, 4 known limits, 8 controls) with the gates' shares under the null, the check against them, the 1.5-fold gate's power and the vehicle-only design; the check's timings; and "known limit" as an outcome, which the documentation tests count. docs/validation_runs.toml records the three runs.
…nding

The permutation check reported the smallest p-value its runs can give from a rule
about renamings of dose groups, which missed the swap of the vehicle and a single
treated dose under the fold-change, Dunnett and Williams-type gates, the reflection
of doses that sit symmetrically on the axis, and renamings drawn among random runs.
Its guard required the true doses to give the result's gate p-values bit for bit,
which another CPU's arithmetic breaks by 1e-13, and its reruns fitted every test of
the result to meet that guard.

The check now counts as ties the runs whose gate gives the true doses' genes,
p-values within a relative 1e-9 and calls, and reports the smallest possible p-value
as (1 + ties) / (1 + runs). The guard compares the same way, each run fits the gate's
test alone with the result's seed, and the check reuses the result's reasons, the
cube's group checks, the gate's wording and the analysis map instead of copies. The
calibration bounds each gate's share of null cell types by the 2,000 cell types it
counts, which makes the dose-group gate's rows at 3 replicates known limits, and
compares the check with the null datasets it did not run on; R counts the ties from
its own refits of every assignment.
The gate's validation recomputed the spline's F-tests that the dose tests' validation
already matches to limma, and its counts of ties recoded the rule they were meant to
check. R now counts as ties the assignments whose refit gives the true doses' genes
and p-values, refits every assignment of the sexes as well as of the duplicate
design, and checks the spline gate's calls on CytePool's p-values; a covariate now
splits the animals' units in R's comparison of units.

The records follow: the gate's script printed 26 checks, 26 pass, and the fixture
shrank from 480 to 406 KB; the contrasts' script printed its recorded table again;
the benchmark puts 100 runs of the spline gate at 340 baselines, within the target
of 350, now that each run fits the gate's test alone.
The documentation said the permutation check shows how often noise passes the gate,
which holds only where no gene responds: in the guide's hepatocytes, 47 of the 51
genes that passed in the runs respond in truth. It now leads with the p-value, which
tests whether the true doses pass more genes than permuted ones, and reads the share
of runs as how often the gate passes genes with the samples' doses shuffled, which
varies between series of one design. The chapter, the glossary and the docstring use
one wording for the gate against BMDExpress's prefilter, the guide splits its two long
cells, and calls are filtered at an adjusted p-value of at most 0.05 throughout.

New tests pin the ties at one treated dose, under Dunnett's test of two doses, for a
spline through unequal groups, for sexes with unequal counts and for symmetric doses,
and that a Monte Carlo gate runs with the result's seed.
The gates' bound counted the 1,000 null datasets of a design cell, while each share
counts its 2,000 cell types, whose standard error between datasets matches the
binomial one, so the bound sat 5.7 standard errors above 5%. At 6.95%, the dose-group
gate exceeds it at 3 replicates too (7.50% and 7.25%), and those rows join the known
limits. The check's runs are compared with the 900 null datasets they did not run
on, at 4 standard errors of the difference, and pass in every design cell.

The run printed 139 checks: 14 pass, 111 information, 6 known limits and 8 controls,
with every other row as the build's run printed it.
…target

The Validation page timed permutation_check against "the baseline of the epic's targets" and said that its runs "lie within the target of 350", terms that the page defines nowhere. The page now names the baseline as it does for multiple_contrast_tests.py, the weighted fit of linear_models.py --benchmark, and says that a check of 100 runs costs about 100 analyses of the cell type. The script's docstring drops the same phrase; docstrings do not count in the recorded digest, so the run's record stays valid.
@parashardhapola
parashardhapola merged commit 24e6c90 into master Nov 21, 2026
6 checks passed
@parashardhapola
parashardhapola deleted the dose/09-responsive-gene-gate branch November 21, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant