Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

## Test, validation and documentation data

The files under `tests/data/` are written by the scripts in `validation/`. Except for the dataset below, their inputs are generated with fixed random seeds. Reference values in those files come from the packages each script names.
The files under `tests/data/` are written by the scripts in `validation/`. Except for the two GEO series below, their inputs are generated with fixed random seeds. Reference values in those files come from the packages each script names.

The pages of the documentation (`docs/source/`) read the datasets that `docs/datasets.py` prepares in a cache outside the repository: three simulated with fixed random seeds by that script, and the two below, GEO series GSE96583 and the MSigDB Hallmark interferon-alpha response set. No file of these datasets is in the repository; the script records the source, licence, citation and checksum of each. Two guides also download the networks CollecTRI and PROGENy, as their sections below say.

Expand All @@ -189,6 +189,14 @@ Kang HM, Subramaniam M, Targ S, Nguyen M, Maliskova L, McCarthy E, Wan E, Wong S

Terms, from NCBI's GEO disclaimer (https://www.ncbi.nlm.nih.gov/geo/info/disclaimer.html): "NCBI places no restrictions on the use or distribution of the GEO data. However, some submitters may claim patent, copyright, or other intellectual property rights in all or a portion of the data they have submitted. NCBI is not in a position to assess the validity of such claims, and therefore cannot provide comment or unrestricted permission concerning the use, copying, or distribution of the information contained in GEO."

### GEO series GSE208161 (Sonigra et al. 2022)

`validation/dose_design.py` downloads `GSE208161_Pt04_Pt11_Pt19_Pt21_integrated_metadata.csv.gz` (1,679,182 bytes, SHA-256 dd36521e006f18416d9c5a0db5741dd374966128bcd9b98510fd0a9c585af4fe), the per-cell metadata of a phase 1 trial, from the Gene Expression Omnibus (https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE208161) into a cache outside the repository, and `tests/data/dose_design/gse208161.npz` holds the cells and summed RNA counts of each participant, day and Seurat cluster derived from it, with each participant's dose from GEO's sample records.

Sonigra A, Nel HJ, Wehr P, Ramnoruth N, Patel S, van Schie KA, Bladen MW, Mehdi AM, Tesiram J, Talekar M, Rossjohn J, Reid HH, Stuurman FE, Roberts H, Vecchio P, Gourley I, Rigby M, Becart S, Toes REM, Scherer HU, Lê Cao KA, Campbell K, Thomas R (2022). Randomized phase I trial of antigen-specific tolerizing immunotherapy with peptide/calcitriol liposomes in ACPA+ rheumatoid arthritis. JCI Insight 7, e160964. doi:10.1172/jci.insight.160964

Terms, from NCBI's GEO disclaimer (https://www.ncbi.nlm.nih.gov/geo/info/disclaimer.html): "NCBI places no restrictions on the use or distribution of the GEO data. However, some submitters may claim patent, copyright, or other intellectual property rights in all or a portion of the data they have submitted. NCBI is not in a position to assess the validity of such claims, and therefore cannot provide comment or unrestricted permission concerning the use, copying, or distribution of the information contained in GEO."

### MSigDB Hallmark gene set HALLMARK_INTERFERON_ALPHA_RESPONSE

The 97 gene symbols of this set (M5911, MSigDB release v2026.2.Hs, from https://www.gsea-msigdb.org/gsea/msigdb/human/geneset/HALLMARK_INTERFERON_ALPHA_RESPONSE) appear in `validation/interferon_stimulated_pbmcs.py` and in `docs/datasets.py`, which writes them for the documentation's pages (`hallmark_interferon_alpha`).
Expand Down
8 changes: 6 additions & 2 deletions cytepool/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@
:func:`meta_analysis`, :class:`MetaAnalysisResult`; replication, :func:`sumrank`,
:class:`SumRankResult`, :func:`replication`, :class:`ReplicationResult`,
:func:`held_out_prediction`, :class:`HeldOutPrediction`.
- Dose response: :class:`DoseAxis`; simulated dose series with known curves,
:func:`simulate_dose_series`, :class:`DoseTruth`.
- Dose response: :class:`DoseAxis`; whether a dose series supports tests in each
cell type, :func:`check_dose_design`, :class:`DoseDesignReport`; simulated dose
series with known curves, :func:`simulate_dose_series`, :class:`DoseTruth`.

The building blocks, which work on arrays, are in ``cytepool.stats`` (linear and
mixed models, empirical Bayes moderation, voom, count models, multiple testing,
Expand Down Expand Up @@ -72,6 +73,7 @@ def _installed_version() -> str:
limma_voom,
)
from .dose_axes import DoseAxis # noqa: E402
from .dose_designs import DoseDesignReport, check_dose_design # noqa: E402
from .gene_set_enrichment import gsea # noqa: E402
from .gene_set_tests import camera, fry # noqa: E402
from .gene_sets import GeneSets, read_gmt, write_gmt # noqa: E402
Expand Down Expand Up @@ -175,6 +177,8 @@ def _installed_version() -> str:
"HeldOutPrediction",
# Dose response.
"DoseAxis",
"check_dose_design",
"DoseDesignReport",
"simulate_dose_series",
"DoseTruth",
# The version.
Expand Down
15 changes: 15 additions & 0 deletions cytepool/_analysis_records.py
Original file line number Diff line number Diff line change
Expand Up @@ -260,3 +260,18 @@ def matched_samples(
positions = np.array([index[value] for value in values], dtype=np.intp)
order = np.argsort(positions, kind="stable")
return positions[order], order


def cube_samples(cube: Cube, design: Design) -> tuple[np.ndarray, np.ndarray]:
"""The position of every design sample in a cube, increasing, and its design row,
with the messages of the workflows that fit every group of a cube."""
return matched_samples(
cube,
design,
no_ids="the design's sample table has no 'sample_id' column, so its rows cannot "
"be matched to the cube's samples; build the Design from cube.samples, or from "
"a table whose 'sample_id' column holds cube sample ids",
text_ids="the cube's does",
data="the cube",
data_ids="the cube's sample ids",
)
81 changes: 76 additions & 5 deletions cytepool/_doses.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
"""The reading of doses that dose axes and simulated dose series share.
"""The reading of doses that dose axes, dose designs and simulated dose series share.

A dose column of a sample table holds real numbers, finite and at least 0, with 0 for
the vehicle. Distinct doses closer than 1e-8 of the larger are refused as one dose
that a conversion of units left as two, such as ``0.1 * 3`` and ``0.3``.
that a conversion of units left as two, such as ``0.1 * 3`` and ``0.3``. A dose axis
names the design columns it builds after the dose column: ``{column}_treated`` and
``{column}_s1``, ``{column}_s2``, ...
"""

from typing import Any
Expand All @@ -11,14 +13,20 @@
import numpy.typing as npt

from ._masked_arrays import Missing, unmasked_array
from ._messages import listing
from ._messages import close_matches, listing
from .table import Table

FloatArray = npt.NDArray[np.float64]

SAME_DOSE = 1e-8
"""Distinct doses closer than this, relative to the larger, are one dose that a
conversion of units left as two."""

LISTED_DOSES = 20
"""The most doses a message lists: dose series rarely have more."""

TREATED_SUFFIX, SPLINE_SUFFIX = "_treated", "_s"


def real_vector(values: Any, name: str, *, missing: Missing = None) -> FloatArray:
"""A new float64 copy of a 1-D array of real numbers."""
Expand All @@ -31,12 +39,75 @@ def real_vector(values: Any, name: str, *, missing: Missing = None) -> FloatArra


def checked_doses(values: Any, name: str) -> FloatArray:
"""Doses as float64: a 1-D array of real numbers, finite and at least 0."""
"""Doses as float64: a 1-D array of real numbers, finite and at least 0, with a
dose of -0.0 read as 0."""
doses = real_vector(values, name)
bad = ~np.isfinite(doses) | (doses < 0.0)
if bad.any():
raise ValueError(
f"{name} must be finite and at least 0, with 0 for the vehicle; got "
f"{listing([repr(float(dose)) for dose in doses[bad]])}"
)
return doses
# -0.0 + 0.0 is 0.0, so that a vehicle given as -0.0 prints as 0.
unsigned: FloatArray = doses + 0.0
return unsigned


def column_doses(table: Table, column: str) -> FloatArray:
"""The checked doses of a sample table's dose column.

Raises:
TypeError: If the column does not hold real numbers.
ValueError: If the table has no such column, or a dose is not finite or is
negative.
"""
if column not in table:
raise ValueError(
f"the dose column {column!r} is not a column of the sample "
f"table{close_matches(column, table.columns)}; columns are: "
f"{listing([repr(name) for name in table.columns])}; name the column "
"that holds the doses with DoseAxis(<column>, ...)"
)
return checked_doses(table[column], f"samples[{column!r}]")


def distinct_doses(doses: FloatArray, unit: str) -> FloatArray:
"""The distinct doses, ascending, refusing two that differ only by rounding.

Raises:
ValueError: If two distinct doses differ by less than ``SAME_DOSE`` of the
larger.
"""
levels = np.unique(doses)
close = np.flatnonzero(np.diff(levels) <= SAME_DOSE * levels[1:])
if close.size:
a, b = float(levels[close[0]]), float(levels[close[0] + 1])
raise ValueError(
f"the doses {a!r} and {b!r} {unit} differ by less than 1e-8 of the dose, "
"as a conversion of units can leave one dose as two; round the dose "
"column first, for example with numpy.round"
)
return levels


def dose_text(doses: FloatArray, unit: str) -> str:
"""Doses for messages, such as "0.1, 1 and 10 ug/kg"."""
return f"{listing([f'{dose:g}' for dose in doses], limit=LISTED_DOSES)} {unit}"


def treated_column(column: str) -> str:
"""The name of the design column that is 1 for a dose above 0."""
return column + TREATED_SUFFIX


def spline_column(column: str, j: int) -> str:
"""The name of the j-th spline column of dose, counted from 1."""
return f"{column}{SPLINE_SUFFIX}{j}"


def is_axis_column(name: str, column: str) -> bool:
"""Whether a variable is the dose column or a column a dose axis builds from it."""
spline = column + SPLINE_SUFFIX
return name in (column, treated_column(column)) or (
name.startswith(spline) and name[len(spline) :].isdigit()
)
38 changes: 28 additions & 10 deletions cytepool/_group_fits.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,21 +146,16 @@ def prepare_group(
checked on the samples used.
"""
cube, settings = inputs.cube, inputs.settings
counts = cube.layer(COUNTS_LAYER, g)[inputs.cube_rows] # design samples, cube order
cells = None if cube.n_cells is None else cube.n_cells[inputs.cube_rows, g]
min_counts = settings["min_counts"]
reasons = left_out(counts, cells, settings["min_cells"], min_counts)
used = np.array([not reason for reason in reasons], dtype=bool)
min_cells, min_counts = settings["min_cells"], settings["min_counts"]
counts, cells, reasons, used = group_samples(
cube, inputs.cube_rows, g, min_cells, min_counts
)
fit = record_type(str(cube.groups["group_id"][g]), used, reasons)
if counts.shape[1] == 0:
return fit.stop("no_features", "the cube has no features"), None
n_used = int(np.count_nonzero(used))
if n_used == 0:
# No counts is fewer than any positive minimum.
rules = f"fewer than {amount(min_counts)} counts" if min_counts else "no counts"
if cells is not None:
rules = f"fewer than {settings['min_cells']} cells, or {rules}"
reason = f"every one of the {used.size} samples was left out ({rules})"
reason = no_samples_reason(used.size, min_cells, min_counts, cells is not None)
return fit.stop("no_samples", reason), None

# The rows of the full design, and the columns that span them.
Expand Down Expand Up @@ -254,6 +249,29 @@ def prepare_group(
return fit, data


def group_samples(
cube: Cube, cube_rows: np.ndarray, g: int, min_cells: int, min_counts: float
) -> tuple[np.ndarray, np.ndarray | None, list[str], np.ndarray]:
"""The counts and cells of the design samples in group g, why each is left out
of the group's fit ("" when used), and whether each is used."""
counts = cube.layer(COUNTS_LAYER, g)[cube_rows] # design samples, cube order
cells = None if cube.n_cells is None else cube.n_cells[cube_rows, g]
reasons = left_out(counts, cells, min_cells, min_counts)
used = np.array([not reason for reason in reasons], dtype=bool)
return counts, cells, reasons, used


def no_samples_reason(
n_samples: int, min_cells: int, min_counts: float, cells_recorded: bool
) -> str:
"""Why a group whose samples were all left out is not fitted."""
# No counts is fewer than any positive minimum.
rules = f"fewer than {amount(min_counts)} counts" if min_counts else "no counts"
if cells_recorded:
rules = f"fewer than {min_cells} cells, or {rules}"
return f"every one of the {n_samples} samples was left out ({rules})"


def left_out(
counts: np.ndarray, cells: np.ndarray | None, min_cells: int, min_counts: float
) -> list[str]:
Expand Down
20 changes: 3 additions & 17 deletions cytepool/differential_expression.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,8 @@

from ._analysis_records import (
cube_record,
cube_samples,
design_record,
matched_samples,
mixed_design_record,
only_key,
software_versions,
Expand Down Expand Up @@ -1338,7 +1338,7 @@ def _analyse(
"prior_weights at None"
)
check_sample_groups(settings, design.samples)
cube_rows, order = _matched(cube, design)
cube_rows, order = cube_samples(cube, design)
tested = _contrasts_to_test(design, contrasts, method)
if settings.get("f_tests"):
_check_f_tests(settings["f_tests"], tested)
Expand Down Expand Up @@ -1389,20 +1389,6 @@ def _analyse(
)


def _matched(cube: Cube, design: Design) -> tuple[np.ndarray, np.ndarray]:
"""The cube position of every design sample, increasing, and its design row."""
return matched_samples(
cube,
design,
no_ids="the design's sample table has no 'sample_id' column, so its rows cannot "
"be matched to the cube's samples; build the Design from cube.samples, or from "
"a table whose 'sample_id' column holds cube sample ids",
text_ids="the cube's does",
data="the cube",
data_ids="the cube's sample ids",
)


def _group_fit(
method: str,
cube: Cube,
Expand All @@ -1413,7 +1399,7 @@ def _group_fit(
entry: Mapping[str, Any],
) -> GroupFit:
"""The fit of one group computed again, as the analysis computed it."""
cube_rows, order = _matched(cube, design)
cube_rows, order = cube_samples(cube, design)
g = group_positions(cube, group)[0]
inputs = GroupInputs(method, cube, order, cube_rows, contrasts, settings)
fit_group = _fit_dream_group if method == "dream_voom" else _fit_group
Expand Down
Loading