Pipeline

The staged orchestrator and its result manifest.

rustscenic.pipeline

End-to-end rustscenic stage orchestrator.

Public API: rustscenic.pipeline.run(rna, output_dir, *, fragments=None, peaks=None, tfs=None, motif_rankings=None, ...) -> PipelineResult

One call runs every rustscenic stage the user provides input for:

1. preproc  (fragments + peaks)      → cells × peaks AnnData
2. topics   (cells × peaks AnnData)  → cell-topic + topic-peak matrices
3. grn      (RNA expression + TFs)   → TF-target importances
4. regulons (grn)                    → top-N targets per TF
5. cistarget (regulons + motif DB)   → motif-enriched regulons [optional]
6. enhancer (RNA + ATAC + TSS)       → peak-gene links [optional]
7. eRegulon (GRN + motifs + links)   → TF-enhancer-gene modules [optional]
8. aucell   (RNA + regulons)         → per-cell regulon activity

Outputs are written to output_dir as parquet / json / h5ad files so downstream notebooks can pick up where the pipeline left off.

No new Python dependencies. Uses only numpy, pandas, pyarrow, scipy, plus the rustscenic Rust backend.

PipelineResult dataclass

PipelineResult(
    output_dir: Path,
    atac_matrix_path: Path | None = None,
    cell_barcode_filter: dict[str, int] | None = None,
    grn_path: Path | None = None,
    regulons_path: Path | None = None,
    candidate_regulons_path: Path | None = None,
    pruned_regulons_path: Path | None = None,
    aucell_path: Path | None = None,
    topics_dir: Path | None = None,
    cistarget_path: Path | None = None,
    enhancer_links_path: Path | None = None,
    eregulons_path: Path | None = None,
    integrated_adata_path: Path | None = None,
    elapsed: dict = dict(),
    io_elapsed: dict = dict(),
    memory: dict = dict(),
    n_cells: int | None = None,
    n_grn_edges: int | None = None,
    n_regulons: int | None = None,
    n_candidate_regulons: int | None = None,
    n_pruned_regulons: int | None = None,
    n_cistarget_rows: int | None = None,
    n_enhancer_links: int | None = None,
    n_eregulon_rows: int | None = None,
    n_eregulons: int | None = None,
    aucell_shape: list[int] | None = None,
    regulon_source: str = "candidate_grn_activator_repressor",
    grn_fit: dict = dict(),
    grn_correlation: dict = dict(),
    topics_fit: dict = dict(),
    backend_execution: dict = dict(),
    matrix_inputs: dict = dict(),
    cistarget_rankings: dict = dict(),
    region_cistarget_rankings: dict = dict(),
)

Artifacts and metadata from a pipeline run.

All file paths point inside output_dir. Stages that were skipped because inputs weren't provided have None for their result path.

Attributes:

Name Type Description
output_dir Path

Directory containing the run's written artifacts and manifest.

atac_matrix_path Path | None

Written cells-by-peaks AnnData file, when ATAC inputs were available.

cell_barcode_filter dict[str, int] | None

Optional barcode-filter counts recovered from ATAC metadata.

grn_path Path | None

Written TF-target importance table.

regulons_path Path | None

Written active regulon collection used for scoring.

candidate_regulons_path Path | None

Written GRN-derived candidate regulons before motif pruning.

pruned_regulons_path Path | None

Written motif-pruned regulons, when annotation pruning was run.

aucell_path Path | None

Written cells-by-regulons activity scores.

topics_dir Path | None

Directory containing the fitted topic matrices, when ATAC was used.

cistarget_path Path | None

Written motif enrichment table, when motif rankings were supplied.

enhancer_links_path Path | None

Written peak-to-gene links, when linking inputs were available.

eregulons_path Path | None

Written long eRegulon edge table, when assembly was run.

integrated_adata_path Path | None

RNA AnnData file with AUCell scores attached, unless writing was disabled.

elapsed dict

Recorded stage elapsed times in seconds.

io_elapsed dict

Separately instrumented IO elapsed times in seconds.

memory dict

Process peak-RSS snapshots observed after recorded stages. These are cumulative process peaks, not incremental allocations per stage.

n_cells int | None

Number of RNA cells entering the run.

n_grn_edges int | None

Number of inferred TF-target edges.

n_regulons int | None

Number of active regulons used for AUCell scoring.

n_candidate_regulons int | None

Number of GRN-derived candidate regulons before motif pruning.

n_pruned_regulons int | None

Number of regulons retained by motif pruning, when performed.

n_cistarget_rows int | None

Number of reported motif enrichment rows, when enrichment was run.

n_enhancer_links int | None

Number of retained peak-to-gene links, when linking was run.

n_eregulon_rows int | None

Number of rows in the assembled long eRegulon edge table.

n_eregulons int | None

Number of assembled eRegulons.

aucell_shape list[int] | None

Activity-matrix dimensions as [cells, regulons].

regulon_source str

Label recording how the active regulon collection was selected.

grn_fit dict

Recorded GRN fitting settings and fitted-tree summaries.

grn_correlation dict

Recorded polarity and correlation settings and edge-count summaries.

topics_fit dict

Recorded topic-model settings and assignment summaries.

backend_execution dict

Per-stage backend symbol provenance or reasons stages were skipped.

matrix_inputs dict

Recorded profiles of the RNA and shared-cell ATAC inputs.

cistarget_rankings dict

Recorded gene-ranking input provenance and projection metadata.

region_cistarget_rankings dict

Recorded region-ranking input provenance and projection metadata.

output_dir instance-attribute

output_dir: Path

atac_matrix_path class-attribute instance-attribute

atac_matrix_path: Path | None

cell_barcode_filter class-attribute instance-attribute

cell_barcode_filter: dict[str, int] | None

grn_path class-attribute instance-attribute

grn_path: Path | None

regulons_path class-attribute instance-attribute

regulons_path: Path | None

candidate_regulons_path class-attribute instance-attribute

candidate_regulons_path: Path | None

pruned_regulons_path class-attribute instance-attribute

pruned_regulons_path: Path | None

aucell_path class-attribute instance-attribute

aucell_path: Path | None

topics_dir class-attribute instance-attribute

topics_dir: Path | None

cistarget_path class-attribute instance-attribute

cistarget_path: Path | None
enhancer_links_path: Path | None

eregulons_path class-attribute instance-attribute

eregulons_path: Path | None

integrated_adata_path class-attribute instance-attribute

integrated_adata_path: Path | None

elapsed class-attribute instance-attribute

elapsed: dict

io_elapsed class-attribute instance-attribute

io_elapsed: dict

memory class-attribute instance-attribute

memory: dict

n_cells class-attribute instance-attribute

n_cells: int | None

n_grn_edges class-attribute instance-attribute

n_grn_edges: int | None

n_regulons class-attribute instance-attribute

n_regulons: int | None

n_candidate_regulons class-attribute instance-attribute

n_candidate_regulons: int | None

n_pruned_regulons class-attribute instance-attribute

n_pruned_regulons: int | None

n_cistarget_rows class-attribute instance-attribute

n_cistarget_rows: int | None
n_enhancer_links: int | None

n_eregulon_rows class-attribute instance-attribute

n_eregulon_rows: int | None

n_eregulons class-attribute instance-attribute

n_eregulons: int | None

aucell_shape class-attribute instance-attribute

aucell_shape: list[int] | None

regulon_source class-attribute instance-attribute

regulon_source: str

grn_fit class-attribute instance-attribute

grn_fit: dict

grn_correlation class-attribute instance-attribute

grn_correlation: dict

topics_fit class-attribute instance-attribute

topics_fit: dict

backend_execution class-attribute instance-attribute

backend_execution: dict

matrix_inputs class-attribute instance-attribute

matrix_inputs: dict

cistarget_rankings class-attribute instance-attribute

cistarget_rankings: dict

region_cistarget_rankings class-attribute instance-attribute

region_cistarget_rankings: dict

manifest

manifest() -> dict

Return result metadata, converting top-level Path values to strings.

run

run(
    rna: str | Path | Any,
    output_dir: str | Path,
    *,
    adata_atac: Any | None = None,
    fragments: str | Path | None = None,
    peaks: str | Path | None = None,
    tfs: str | Path | Iterable[str] | None = None,
    motif_rankings: str | Path | DataFrame | None = None,
    motif_annotations: str | Path | DataFrame | None = None,
    region_motif_rankings: str | Path | DataFrame | None = None,
    gene_coords: str | Path | DataFrame | None = None,
    grn_n_estimators: int = 500,
    grn_max_features: float = 0.1,
    grn_early_stop_window: int = 25,
    grn_early_stop_mode: str = "arboreto",
    grn_target_block_size: int | None = None,
    grn_top_targets: int = 50,
    grn_regulon_polarities: str = "both",
    grn_rho_threshold: float = 0.03,
    grn_rho_mask_dropouts: bool = False,
    aucell_top_frac: float = 0.05,
    topics_n_topics: int = 30,
    topics_n_passes: int = 3,
    topics_method: str = "vb",
    topics_n_iters: int = 200,
    topics_n_threads: int = 1,
    cistarget_top_frac: float = 0.05,
    cistarget_auc_threshold: float = 0.05,
    cistarget_nes_threshold: float | None = None,
    enhancer_max_distance: int = 500000,
    enhancer_min_abs_corr: float = 0.1,
    eregulon_min_target_genes: int = 5,
    eregulon_min_enhancer_links: int = 2,
    write_integrated_adata: bool = True,
    seed: int = 777,
    verbose: bool = True,
) -> PipelineResult

Run the available rustscenic stages end-to-end.

The workflow runs only the stages the user supplies inputs for. At minimum, rna is required (for GRN + AUCell). Providing fragments and peaks enables preproc + topics. Providing motif_rankings enables cistarget.

Parameters:

Name Type Description Default
rna str | Path | Any

An AnnData, a path to an .h5ad, or a pandas DataFrame (cells × genes).

required
output_dir str | Path

Directory where all artifacts are written. Created if missing.

required
adata_atac Any | None

Pre-built cells × peaks AnnData, or a path to one on disk. Use this when you already have a cleaned/subset ATAC matrix (e.g. cell-called barcodes only, post-QC). Mutually exclusive with fragments + peaks: if adata_atac is provided, the fragments + peaks path is skipped. This avoids carrying the full raw 10x barcode set (~450k empty droplets typical) through topics, which can stall downstream stages on consumer hardware.

None
fragments str | Path | None

Paths to a 10x-style fragments.tsv[.gz] and peak BED. When both are provided AND adata_atac is not, rustscenic.preproc builds the cells × peaks AnnData and topics fits on it.

None
peaks str | Path | None

Paths to a 10x-style fragments.tsv[.gz] and peak BED. When both are provided AND adata_atac is not, rustscenic.preproc builds the cells × peaks AnnData and topics fits on it.

None
tfs str | Path | Iterable[str] | None

Candidate transcription factor names. Path to a newline-separated file, an iterable of strings, or None to use the bundled human TF list.

None
motif_rankings str | Path | DataFrame | None

Motif ranking DataFrame, or a path to a parquet / feather file with motifs as rows and genes as columns. If provided, cistarget runs to score candidate regulons for motif enrichment.

None
motif_annotations str | Path | DataFrame | None

Optional motif-to-TF annotation table. When provided alongside motif_rankings, the active regulons are pruned to enriched motifs annotated back to the source TF, and target genes are restricted to the motif ranking recovery window.

None
region_motif_rankings str | Path | DataFrame | None

Optional region-based motif ranking DataFrame, or path, with motifs as rows and peak / region IDs as columns. When supplied alongside ATAC inputs and gene coordinates, eRegulon assembly uses this exact region-cistarget path instead of the gene-cistarget bridge. File-backed parquet / feather rankings are projected to the peaks used by the current run so large region-ranking databases do not need to be loaded in full.

None
gene_coords str | Path | DataFrame | None

DataFrame with columns ['gene', 'chrom', 'tss'], or a path to a parquet/csv file with the same shape. When supplied alongside fragments + peaks, the orchestrator runs rustscenic.enhancer.link_peaks_to_genes and, when either gene- or region-based motif rankings are supplied, rustscenic.eregulon.build_eregulons.

None
topics_method str

"vb" (default) - online VB LDA, fast at small K (≤ 10). "gibbs" - collapsed-Gibbs LDA (Mallet-class), slower per sweep but recovers ~10× more distinct topics on sparse scATAC at K ≥ 30. Pair with topics_n_threads > 1 for AD-LDA parallel execution at atlas scale. Keep the thread count fixed across biological comparisons because it changes the AD-LDA partition and may change the fitted posterior mode.

'vb'
topics_n_iters int

Gibbs sweeps (only used when topics_method='gibbs'). 200 is a reasonable default; bump to 500–1000 for higher-quality posterior estimates.

200
topics_n_threads int

Threads for the Gibbs sampler (only used when topics_method='gibbs'). 1 = bit-deterministic serial path. > 1 = AD-LDA parallel path, reproducible at a fixed seed and thread count. Treat the thread count as part of the model configuration, not only as a scheduler setting.

1
grn_max_features float

Fraction of candidate TFs sampled per split in GRN boosting. 0.1 matches arboreto/SCENIC defaults. Lower values can be much faster on high-TF datasets, but change edge rankings and should be treated as a speed/quality tradeoff.

0.1
grn_early_stop_mode str

"arboreto" (default) uses the trailing mean of OOB improvements. "legacy_inbag" reproduces historical RustScenic early stopping.

'arboreto'
grn_regulon_polarities str

"both" (default) computes TF-target Pearson correlations and emits separate <TF>_activator and <TF>_repressor regulons. "activating" keeps only positive-correlation regulons. "unsigned" is the explicit legacy path and skips dichotomisation.

'both'
grn_target_block_size int | None

Retained for API compatibility. Flat GRN scheduling ignores the value.

None
grn_n_estimators int

Maximum boosting trees fitted per target gene. Early stopping may fit fewer; passed to grn.infer.

500
grn_early_stop_window int

Trailing-window size for the selected GRN stopping monitor. Zero disables early stopping.

25
grn_top_targets int

Maximum highest-importance targets per candidate regulon, applied separately to each TF and polarity in signed mode. Candidate sets require at least min(10, grn_top_targets) genes.

50
grn_rho_threshold float

Absolute Pearson-correlation threshold for activating or repressing GRN edges. Neutral edges are excluded from signed regulons. Ignored for grn_regulon_polarities="unsigned".

0.03
grn_rho_mask_dropouts bool

Exclude cells where either gene is zero when scoring correlations. Ignored for unsigned regulons.

False
aucell_top_frac float

Fraction of the highest-ranked genes per cell used for AUCell recovery, greater than zero and at most one.

0.05
topics_n_topics int

Number of latent topics requested when ATAC input is available.

30
topics_n_passes int

Passes over the ATAC corpus for topics_method="vb". Gibbs fitting uses topics_n_iters instead.

3
cistarget_top_frac float

Ranking-universe fraction used for enrichment recovery AUC and for the target-recovery window during motif-annotation pruning.

0.05
cistarget_auc_threshold float

Minimum motif AUC for enrichment reporting and annotation-based motif filtering on the gene- and region-ranking paths.

0.05
cistarget_nes_threshold float | None

Optional minimum normalized enrichment score. None applies no NES filter; a supplied threshold excludes undefined NES values.

None
enhancer_max_distance int

Maximum peak-center to gene-TSS distance in base pairs for enhancer-to-gene linking when its inputs are available.

500000
enhancer_min_abs_corr float

Minimum absolute correlation retained for enhancer-to-gene links.

0.1
eregulon_min_target_genes int

Minimum surviving target genes required for an assembled eRegulon.

5
eregulon_min_enhancer_links int

Minimum peak-to-gene links required for an assembled eRegulon.

2
write_integrated_adata bool

Write rna_with_regulons.h5ad with AUCell scores attached to obs. Default True preserves the full end-to-end artifact. Set False for HPC profiling runs that need to measure compute stages without the final full-RNA h5ad materialisation and write.

True
seed int

Random seed passed to GRN inference and the selected topic fitter. Hold the Gibbs thread count and other settings fixed as well.

777
verbose bool

Enable pipeline progress logging. Called stages may still emit their own messages or warnings when this is false.

True

Returns:

Type Description
PipelineResult - dataclass with paths to every artifact written.