V1 prep

[parashardhapola]

Scarf 1.0 is a storage and I/O rewrite for reliable local and cloud analysis at atlas scale, plus API hardening, mapping/reference tooling, and a much larger test surface. Existing Zarr v2 datasets remain fully usable. Most day-to-day DataStore workflows keep the same call sites. Migration cost is mainly environment (Python 3.12+), Dask removal, and a small set of renamed or clarified APIs.

---

Migration guide (start here)

1. Environment

	0.32.x	1.0.0
Python	≥3.11 (docs often said 3.10+)	≥3.12 (3.12 / 3.13 / 3.14 tested)
Packaging	requirements.txt + setup.py metadata	PEP 621 pyproject.toml
Install	pip install scarf[extra]	uv pip install scarf[extra] (pip still works)

Removed dependencies: dask, polars, pyarrow, joblib, importlib_metadata, gensim (extra), cmocean (extra).

Major upgrades: zarr<=2.16 → zarr>=3.1.2, numpy==1.26 → numpy>=2, numcodecs>=0.16, plus obstore for object-store URIs.

conda create -n scarf_v1 python=3.12
conda activate scarf_v1
uv pip install 'scarf[extra]'

2. Existing Zarr stores (no conversion required)

Open your current .zarr with Scarf 1.0 as usual. Zarr Python 3 reads v2 layouts; Scarf wraps arrays with ChunkedArray instead of Dask.

• Analysis, filtering, graphs, embeddings, markers (legacy layout), and mapping all work on existing stores.
• Writing new metadata or derived arrays into a v2 store keeps v2-compatible codecs and layout. Scarf does not force an in-place v2→v3 rewrite of your counts.
• Sharding finalization (finalize_sharded_counts) no-ops on v2 groups.

Optional performance upgrade only: if you want a new Zarr v3 copy with sharded counts (better local throughput or object-store IO), use:

uv run python -m scarf.tools.repack_zarr old.zarr new.zarr --profile fast_local
# cloud-oriented compressors / layout:
uv run python -m scarf.tools.repack_zarr old.zarr new.zarr --profile cloud

from scarf.tools.repack_zarr import repack_store
repack_store("old.zarr", "new.zarr", profile="fast_local", shard_counts=True)

This writes a separate output store. It is not part of the upgrade path and is unnecessary for users who already hold large archives and are happy with current IO.

New ingest (CrToZarr, H5adToZarr, merge, etc.) writes Zarr v3 and finalizes sharded counts automatically. That applies to newly created datasets, not to upgrading existing ones.

3. Replace Dask usage

Assay.rawData, normed(), and save_normalized_data() return scarf.chunked.ChunkedArray, not dask.array.Array.

# before
result = ds.RNA.rawData.sum(axis=0).compute()

# after
from scarf.utils import controlled_compute
result = controlled_compute(ds.RNA.rawData.sum(axis=0), ds.nthreads)
# or
result = ds.RNA.rawData.sum(axis=0).compute(nthreads=ds.nthreads)

Custom normalization callables must be (Assay, ChunkedArray) -> ChunkedArray.

Names kept for compatibility (no Dask underneath): dask_to_zarr, show_dask_progress.

4. Imports: no more star exports

scarf/__init__.py no longer does from .readers import * (and similar). Use the explicit public surface or submodule imports.

import scarf
from scarf import DataStore, CrH5Reader, CrToZarr, AssayMerge

from scarf.mapping_reference import MappingReference, MappingResult
from scarf.harmony import run_harmony, fit_harmony, HarmonyResult
from scarf.metrics import compute_lisi, label_concordance_score, lisi_batch_mixing_score

Notable removals:

• MetaData.to_polars_dataframe (polars removed; use to_pandas_dataframe)
• Relying on from scarf import * for every submodule helper

5. Cloud and memory knobs (new, optional)

ds = scarf.DataStore(
    "s3://bucket/dataset.zarr",
    storage_options={...},      # passed to obstore
    zarrProfile="cloud",        # or "fast_local"; env: SCARF_ZARR_PROFILE
    mem_budget="8G",            # or "0.6" (fraction of RAM); env: SCARF_MEM_BUDGET
    working_copies=8,           # env: SCARF_WORKING_COPIES
    nthreads=8,                 # also drives IO concurrency; env: SCARF_WORKERS
)

Writers (CrToZarr, H5adToZarr, DatasetMerge, …) accept the same storage_options / mem_budget / working_copies pattern.

On Zarr v3 writers, chunk_size=(1000, 1000) remains in the signature but is largely advisory: geometry comes from the memory budget. On existing v2 stores, prior chunking is left alone.

6. API renames and clarifications

Old	New / action
ZarrMerge	Prefer AssayMerge (ZarrMerge still works, warns)
metric_integration(...)	Prefer metric_label_concordance(...) (same ARI/NMI between two label columns). For neighborhood batch mixing use metric_batch_mixing(...)
metrics.integration_score	Prefer metrics.label_concordance_score
run_mapping(..., exclude_missing=True)	Prefer missing_feature_policy="intersection"
run_mapping(..., run_coral=True)	Deprecated; blocked with harmonized / Symphony references
ref_mu / ref_sigma	Deprecated and ignored; mapping always uses reference stats
Heatmap default cmap cmocean.cm.matter_r	Default is magma_r (cmocean removed)

7. Mapping defaults that can change results

Several mapping helpers now resolve cell_key=None via the latest graph keys instead of hard-coding "I":

• run_mapping, get_mapping_score, get_target_classes
• run_unified_umap, run_unified_tsne, load_unified_graph

If you relied on implicit "I" while your latest graph used another cell key, pass cell_key="I" explicitly.

missing_feature_policy for ordinary mapping: "zero" (default when unset), "intersection", or "error". Harmonized / MappingReference paths additionally support "reference_mean" (their neutral default).

8. When to recompute (correctness, not storage format)

You do not need to rebuild stores to upgrade. Recompute only if you depend on these fixed or redesigned paths:

1. WNN / multimodal integration (integrate_assays(..., method="wnn")) – internal weighting and k handling were rewritten. Old WNN graphs still load; rebuild if you want the corrected behavior.
2. Marker search – new writes use compact_v2. Legacy marker tables still read. Re-run run_marker_search only if you want the new layout.
3. Harmonized atlas mapping – prefer build_mapping_reference then run_mapping / MappingReference.map_query. Legacy graphs without a reference artifact may auto-rebuild once (needs zarr_mode="r+").
4. Newly written normalized matrices – Scarf now writes float32 (was float64). Existing on-disk normed arrays are unchanged.
5. ANN indices – new saves go in-Zarr as ann_idx_bytes. Legacy filesystem ann_idx is still found and can be migrated on read.

9. Signature / default gotchas

# mark_hvgs: max_cells default is None (unlimited), not np.inf
ds.mark_hvgs(..., max_cells=None)

# run_marker_search: gene_batch_size=None auto-sizes from Zarr column layout
ds.run_marker_search(..., gene_batch_size=None)

# make_graph: optional local staging for remote stores; Harmony kwargs
ds.make_graph(..., local_cache="auto", harmony_params={...})

# run_pseudotime_scoring: disconnected graphs
ds.run_pseudotime_scoring(..., component_policy="largest")  # or "error"

# run_pseudotime_aggregation: nan_cluster_value must be int (str no longer allowed)
ds.run_pseudotime_aggregation(..., nan_cluster_value=-1)

# metric_lisi / metric_silhouette gained optional kwargs
ds.metric_lisi(..., perplexity=30)
ds.metric_silhouette(..., random_seed=4444, sample_size=11)

load_zarr(..., synchronizer=...) still accepts synchronizer but ignores it under Zarr v3 (debug log only). Multi-process concurrent writes via ThreadSynchronizer are not on the supported path.

ZARRLOC is now str | zarr.abc.store.Store (no zarr.LRUStoreCache).

Suggested upgrade path

1. New Python 3.12+ env; install scarf[extra].
2. Open an existing store; smoke-test filter_cells → mark_hvgs → make_graph → run_umap.
3. Replace any Dask / polars / ZarrMerge / metric_integration-as-batch-mixing usage.
4. Rebuild WNN only if you use multimodal integration and want the fixed weights.
5. For batch-corrected atlas mapping, adopt build_mapping_reference; use metric_batch_mixing for batch QC.
6. Optionally, if cloud or shard-aligned IO would help, create a new store with repack_zarr. Skip this for large archives unless you have a measured need.

---

Breaking changes (checklist)

1. Python 3.12+ required.
2. Dask removed; lazy arrays are ChunkedArray.
3. New writes use Zarr v3 with budget-driven sharding. Existing v2 stores remain supported without conversion.
4. Newly written normalized matrices use float32 (existing float64 normed arrays unchanged).
5. Polars / pyarrow / cmocean / gensim removed from the dependency set.
6. Explicit package exports (no star-import dump).
7. MetaData.to_polars_dataframe removed.
8. On v3 writers, chunk_size no longer owns count/norm geometry (memory budget does).
9. Mapping cell_key defaults and feature-missing policy semantics (see above).
10. Metrics naming: batch mixing vs label concordance are separate APIs.

---

Behavioural changes (same API, different outcomes)

• Memory-budget layout (new v3 writes): count matrices use sharded geometry from RAM and working_copies (default 8). Normed matrices use full-width row chunks, no shards.
• Compressors (new v3 writes): fast_local → Blosc/LZ4+bitshuffle; cloud → Zstd. Env SCARF_ZARR_PROFILE.
• Shard-parallel IO: ordered shard processing with small read-ahead when shards exist; BLAS threads pinned to 1 per shard for reproducibility.
• Graph build local_cache="auto": remote stores stage the normed matrix locally before PCA/ANN/kNN.
• WNN: new affinity weighting (log-sum-exp), min(k1,k2), finite checks.
• Pseudotime: largest connected component by default; stricter source/sink / ss_vec validation; numeric finite regressor checks for pseudotime markers.
• Meld assay: sparse CSC peak↔feature mapping; chromosome iteration fixed; optional peaks_coords skips BED parse.
• Merge: feature ordering via pandas (not polars); coordinated row plans across assays; optional storage_options.
• Markers: compact shared feature index + dense stats on new writes; sort_marker_results on load; optional prefetch. Legacy marker Zarr trees still load.
• Plots: heatmap default cmap magma_r.
• metric_integration: documentation and deprecation clarify it was always label concordance (ARI/NMI), not batch mixing.

---

New features

Cloud-native Zarr

Open and write s3:// / gs:// (and other obstore URLs) through DataStore and writers with storage_options. Remote-aware column-block pipelines and local staging for normed matrices.

Mapping reference + Symphony-style query correction

ref = ds.build_mapping_reference(
    from_assay="RNA", feat_key="hvgs", batch_columns=["batch", "donor"]
)
ref = ds.get_mapping_reference("RNA", "I", "hvgs")
result = ref.map_query(query_assay, "query1", "normed", query_batches=query_meta[["batch"]])

ds.run_mapping(
    target_assay=query_assay,
    target_name="query1",
    target_feat_key="normed",
    query_batches=query_meta[["batch"]],
)

Also new:

• get_target_label_evidence (vote fractions, entropy, conformal sets)
• calibrate_label_transfer_threshold
• project_mapping_layout (place query on a fixed reference UMAP/tSNE)

Doublet detection

ds.run_doublet_detection(cluster_key="RNA_leiden_cluster", label="doublet_score")

Simulates doublets, maps them, scores cells, optionally smooths on the graph.

Integration metrics

ds.metric_batch_mixing("batch")
ds.metric_label_concordance(["predicted", "truth"], metric="ari")
ds.metric_lisi(["batch"], perplexity=30)

Harmony API surface

Documented run_harmony, fit_harmony, HarmonyResult; make_graph(..., harmony_params=...) for graph builds.

Optional repack tool

python -m scarf.tools.repack_zarr creates a new v3 store with optional count sharding. Use when you want performance characteristics of the new layout; not required to keep using existing data.

---

Under the hood (for maintainers)

• New modules: scarf.storage.zarr_store, scarf.storage.budget, scarf.chunked, scarf.parallel, scarf.symphony, scarf.mapping_reference, scarf.doublet_utils, scarf.tools.repack_zarr, scarf._types.
• Single memory-budget rule drives chunk/shard sizes for new count and normed arrays; writers call finalize_writer_counts after ingest on v3 groups (skipped on v2).
• Sparse ingest accumulates into shard-aligned flushes when sharding applies.
• ANN indices live in-store; graph paths can reuse cached AnnStream for mapping/metrics.
• Packaging modernized (PEP 621, dynamic version from VERSION, optional dependency groups extra / test / docs, uv lockfile).
• Typing pass across public APIs; mypy enabled for scarf/.
• Docs: install guide, integration guide, glossary, quickstart, reference-atlas mapping vignette, graph imputation vignette; vignettes pre-executed for RTD.

---

Bug fixes and impact

Area	What went wrong	User impact	Fix
WNN	Fragile affinity math, assumed equal k, weak validation	Wrong multimodal weights / NaNs	Full rewrite with validation and stable weighting
Pseudotime	Disconnected graphs broke PBA	Crash or nonsense scores	component_policy; component handling
Pseudotime markers	Weak regressor checks	Misleading markers	Finite / uniqueness / optional __valid mask
Meld assay	Chromosome / dense cross-map bugs	Wrong peak–gene aggregation on some chromosomes	Sparse CSC mapping + corrected iteration
Mapping projections	Incomplete writes readable	Corrupt projection layouts	Schema v2 + complete flag
Harmonized mapping	Missing portable reference artifact	Non-reproducible / failing maps	MappingReference persistence + auto rebuild
Marker batches	Fixed batch size vs Zarr chunks	Slow or awkward IO	Auto gene_batch_size
Large feature matrices	Blosc / size limits	Failures on very wide matrices	Budget layout + Blosc limit handling
HNSW on 3.12+	Native wheel / build issues	ANN failures on modern Python	HNSWLIB_NO_NATIVE / build pins in CI and uv
Merge	Feature order / row randomization edge cases	Misaligned merged matrices	Shared row plan, chunk validation
Metrics naming	metric_integration sounded like batch mixing	Misleading QC decisions	Split into concordance vs batch-mixing APIs
Logging / RTD / CI	Flaky docs and tests	Broken builds	Fixture download, cached ATAC outputs, vignette execution helpers

---

Tests, coverage, and CI

Tests: ~9 modules under scarf/tests → ~40 under top-level tests/ (~9.8k lines of test code). New coverage includes Zarr store / budget / chunked / parallel / prefetch, ephemeral datastore, graph coverage and cache, mapping (label transfer, reference, Symphony golden JSON), merger, markers, metrics, meld, readers/writers, doublet paths, pseudotime, UMAP/Harmony, knn utils, renorm subset, and repack.

Fixtures are downloaded in CI (tests.download_fixtures) instead of shipping large binaries in-repo.

CI (pytest.yml):

• Python 3.12, 3.13, 3.14 (+ lowest-direct resolution on 3.12)
• Parallel pytest (pytest-xdist)
• Coverage on scarf → Codecov
• Ruff lint/format + mypy on the 3.12 locked job
• Integration tests marked and skipped by default (-m 'not integration')

---

Documentation and packaging

• Root README.md (replaces RST / pypi README split)
• Install docs: Python 3.12+, uv, Zarr v2 compatibility note, optional repack_zarr, mem_budget
• Expanded api.md (MappingReference, Harmony, metrics, merge, meld, utilities, plots)
• New vignettes: reference atlas mapping, graph imputation; refreshed merging guide
• Version: 1.0.0 (was 0.32.3)

---

Deprecations (still work in 1.0)

• ZarrMerge → use AssayMerge
• DataStore.metric_integration → metric_label_concordance or metric_batch_mixing
• metrics.integration_score → label_concordance_score
• run_mapping(exclude_missing=...), run_coral=True, ref_mu / ref_sigma
• Function names dask_to_zarr / show_dask_progress (Dask-free implementations)

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.