Add a resource management workflow and integrate prepared resources into sftool run

[jpflorido]

Add a resource management workflow and integrate prepared resources into sftool run

Description

Introduce an extensible resource-management command group:

sftool resources setup

The command prepares the persistent datasets and generated resources required by SFtool before sample analysis. Once the workflow is fully integrated, sftool run must consume already installed resources instead of downloading or generating shared resources at runtime.

The initial implementation includes only setup. Future commands such as resources check or resources update remain outside the scope of this issue.

The runtime integration should follow the actual SFtool execution stages. Resource preparation belongs exclusively to sftool resources setup; sftool run should only validate, resolve, select, and consume installed resources.

Main requirements

sftool resources setup must:

• load the resource versions bundled with the installed SFtool release;
• download the required external datasets;
• generate PR and RR catalog resources for GRCh37 and GRCh38;
• prepare ClinVar resources for every supported evidence level;
• download HPO and PharmCAT support resources;
• optionally download and index reference genomes;
• write an installed resource manifest;
• avoid repeating downloads or processing when valid outputs already exist.

sftool run must:

• load and validate the installed resource manifest during bootstrap;
• resolve the resources required by the selected assembly, categories, evidence level, and execution modes;
• select the correct catalog BED convention independently for each sample;
• consume the prepared catalog, ClinVar, HPO, PharmCAT, RR-STR, and reference resources;
• never download or generate persistent shared resources at runtime.

The implementation should remain simple and dictionary-based. A generic provider framework, Pydantic models, dataclasses, or a large resource-management hierarchy are not required.

---

Proposed command

sftool resources setup \
    --output-dir /path/to/sftool_resources \
    --resource-version bundled \
    [--download-reference-genomes]

--output-dir

Root directory where installed resources are stored.

--resource-version

Initially support:

bundled
latest

Default:

bundled

Bundled mode uses dataset versions pinned for the installed SFtool release.

--download-reference-genomes

When enabled, download and index the GRCh37 and GRCh38 FASTA files defined in the bundled resource specification.

---

Bundled resource specification

Add and package:

sftool/data/resources/bundled_resources.json

This file defines the versions, URLs, filenames, and checksums associated with an SFtool release. It must include definitions for:

• ClinVar;
• HPO gene-to-phenotype data;
• PharmCAT positions;
• GRCh37 and GRCh38 reference genomes.

The file must be loaded using importlib.resources and must not depend on the current working directory.

---

Expected resource structure

<output-dir>/
├── catalogs/
│   ├── GRCh37/
│   ├── GRCh38/
│   └── RR_STR/
├── clinvar/
│   ├── source/
│   ├── GRCh37/
│   └── GRCh38/
├── hpo/
├── pharmcat/
├── references/
│   ├── GRCh37/
│   └── GRCh38/
└── resources.json

---

Catalog resources

Generate PR and RR resources for both assemblies:

catalogs/GRCh37/PR.bed
catalogs/GRCh37/PR.chr.bed
catalogs/GRCh37/PR.json
catalogs/GRCh37/RR.bed
catalogs/GRCh37/RR.chr.bed
catalogs/GRCh37/RR.json

catalogs/GRCh38/PR.bed
catalogs/GRCh38/PR.chr.bed
catalogs/GRCh38/PR.json
catalogs/GRCh38/RR.bed
catalogs/GRCh38/RR.chr.bed
catalogs/GRCh38/RR.json

Coordinates must be obtained from the appropriate Ensembl endpoint and reused to write both chromosome naming conventions.

Preserve the existing GRCh37 aliases:

MMUT  -> MUT
ELP1  -> IKBKAP
G6PC1 -> G6PC
GBA1  -> GBA

The RR-STR catalog remains a bundled CSV definition. No RR-STR BED or JSON resources are generated.

---

ClinVar resources

ClinVar preparation is divided into two tasks while retaining the original Task 8 for backward compatibility.

Task 8a — Generate assembly-level ClinVar databases

This corresponds to the original completed Task 8 — Generate ClinVar databases for both assemblies.

Download the pinned ClinVar variant_summary and submission_summary files and generate the intermediate databases for:

GRCh37
GRCh38

The downloaded source files and assembly-level intermediate databases must be retained.

Task 8b — Generate catalog- and evidence-specific ClinVar resources

Using the assembly-level databases and the generated PR/RR catalogs, generate cumulative ClinVar JSON resources for every combination of:

assembly: GRCh37, GRCh38
catalog:  PR, RR
evidence: 1, 2, 3, 4

This produces 16 directly consumable resources:

2 assemblies × 2 catalogs × 4 evidence levels

Evidence levels are cumulative. For example, evidence level 2 includes variants meeting level 2 or higher.

The transformation must:

• parse submission_summary only once;
• use the generated PR and RR catalog JSON files as gene sets;
• filter records by assembly and catalog genes;
• retain the ClinVar variant types supported by SFtool;
• discard records without valid VCF-compatible coordinates;
• preserve the fields required by the current SFtool ClinVar workflow;
• register every generated resource in the installed manifest.

Suggested implementation entry point:

build_clinvar_catalog_resources(...)

Task 8b prepares and installs the resources. Runtime selection is handled later during bootstrap and workflow integration.

---

HPO and PharmCAT resources

Download, without additional transformation:

hpo/genes_to_phenotype_<version>.txt
pharmcat/pharmcat_positions_<version>.vcf

Use stable installed paths and register the files and versions in the manifest.

The PharmCAT executable and its runtime dependencies remain configured separately. Only the PharmCAT support resource belongs to the installed resource bundle.

---

Reference genomes

When --download-reference-genomes is enabled:

• download GRCh37 and GRCh38;
• validate the FASTA files;
• store each one as genome.fa;
• generate genome.fa.fai;
• register the installed paths in the manifest.

Expected structure:

references/GRCh37/genome.fa
references/GRCh37/genome.fa.fai
references/GRCh38/genome.fa
references/GRCh38/genome.fa.fai

Explicit user-provided reference genome paths remain supported as runtime overrides.

---

Installed manifest

Write:

<output-dir>/resources.json

The manifest must contain:

• schema version;
• selected resource channel;
• SFtool version;
• resolved dataset versions;
• installed relative paths;
• SHA-256 checksums;
• catalog resources by assembly;
• ClinVar source files;
• ClinVar assembly-level databases;
• ClinVar resources by assembly, catalog, and evidence level;
• HPO and PharmCAT paths;
• RR-STR catalog path;
• optional reference genome paths;
• installation metadata.

The schema should remain simple and dictionary-based. Paths stored in the manifest must be relative to <output-dir> so that the complete resource directory can be moved without rewriting every entry.

A valid manifest must expose runtime-consumable resources unambiguously. The current implementation may use structures such as:

datasets.catalogs.assemblies.<assembly>.<catalog>.bed
datasets.catalogs.assemblies.<assembly>.<catalog>.chr_bed
datasets.catalogs.assemblies.<assembly>.<catalog>.json
datasets.catalogs.RR_STR.csv

datasets.clinvar.databases.<assembly>
datasets.clinvar.filtered_databases.<assembly>.<catalog>.<evidence>

datasets.hpo.file
datasets.pharmcat.positions_vcf

datasets.reference_genomes.<assembly>.fasta
datasets.reference_genomes.<assembly>.fai

The reference_genomes entries may be omitted, represented as null, or contain null assembly entries when setup is executed without --download-reference-genomes.

---

Idempotency and validation

Setup must:

• reuse existing valid files;
• use temporary .part files for downloads;
• atomically rename completed downloads;
• validate required outputs;
• reject incomplete or inconsistent installations;
• avoid silently overwriting a manifest that refers to incompatible resources.

Runtime bootstrap must:

• validate manifest JSON and schema compatibility;
• resolve the installation root;
• reject absolute paths or paths escaping the resource root;
• verify that required files exist;
• verify checksums;
• fail early with actionable error messages;
• validate only supported assemblies, catalogs, and evidence levels;
• resolve a configured reference genome override before falling back to an installed reference.

---

Implementation phases and task checklist

The original task numbering and completion state up to Task 14 are retained. Tasks from 15 onwards are reorganized around the actual sftool run execution stages and runtime responsibilities.

Phase 1 — Resource CLI foundation and bundled specification

• Task 1 — Add the sftool resources command group

  • Add the extensible Click command group.
  • Register the initial resources setup subcommand.

• Task 2 — Add the bundled resource specification

  • Add and package sftool/data/resources/bundled_resources.json.
  • Define pinned versions, URLs, filenames, and checksums.
  • Load the file through importlib.resources.

• Task 3 — Add basic resource utility functions

  • Add small shared helpers for downloads, checksums, manifest loading/writing, and installed-file validation.
  • Keep orchestration in the command implementation.

Phase 2 — Catalog preparation

• Task 4 — Decouple catalog generation from patient VCFs

  • Remove the dependency on a sample VCF when retrieving catalog coordinates.
  • Generate reusable resources directly from the bundled catalog definitions.

• Task 5 — Generate both BED chromosome conventions

  • Generate BED files with 1, 2, X, MT naming.
  • Generate equivalent .chr.bed files with chr1, chr2, chrX, chrM naming.
  • Retrieve coordinates only once per catalog and assembly.

• Task 6 — Generate all catalog resources during setup

  • Generate PR and RR BED/JSON resources for GRCh37 and GRCh38.
  • Preserve existing GRCh37 aliases.
  • Retain RR-STR as its bundled CSV definition without generating BED or JSON files.

Phase 3 — External datasets and installed manifest

• Task 7 — Download the bundled ClinVar snapshot

  • Download the pinned variant_summary and submission_summary files.
  • Preserve the downloaded source files under clinvar/source/.

• Task 8a — Generate ClinVar databases for both assemblies

  • This is the original completed Task 8 and retains its historical status.
  • Generate the assembly-level ClinVar intermediate databases for GRCh37 and GRCh38.

• Task 8b — Generate catalog- and evidence-specific ClinVar JSON resources

  • Generate resources for PR and RR.
  • Generate them for GRCh37 and GRCh38.
  • Generate cumulative evidence levels 1, 2, 3, and 4.
  • Produce and register all 16 resulting JSON resources.
  • Parse shared ClinVar inputs only once where possible.

• Task 9 — Download HPO and PharmCAT resources

  • Download the pinned HPO gene-to-phenotype file.
  • Download the pinned PharmCAT positions VCF.
  • Store both at stable installed paths.

• Task 10 — Add optional reference genome download

  • Download GRCh37 and GRCh38 only when requested.
  • Validate, normalize filenames, and generate FASTA indexes.

• Task 11 — Write the installed resource manifest

  • Generate resources.json.
  • Register versions, paths, checksums, and installation metadata.
  • Include the Task 8b resources indexed by assembly, catalog, and evidence level.

Phase 4 — Bootstrap and runtime resource resolution

• Task 12 — Add the manifest to runtime configuration

  • Allow runtime configuration to locate the installed resource bundle and manifest.
  • Keep executable and runtime-tool configuration separate from installed resource data.

• Task 13 — Validate the resource bundle during bootstrap

  • Verify manifest compatibility and required installed files before analysis starts.
  • Verify checksums and fail early with actionable error messages.

• Task 14 — Resolve shared execution resources

  • Resolve HPO, PharmCAT, RR-STR, ClinVar, catalogs, and reference genomes from the manifest.
  • Select resources for the requested assembly, requested categories, and ClinVar evidence level.
  • Preserve explicit user-provided reference genome overrides.
  • Expose resolved paths through a runtime resource object attached to the execution context.

Phase 5 — sftool run stage integration

• Task 15 — Complete the runtime resource access API

  • Finalize the RuntimeResources public API.
  • Expose catalog JSON resources by category.
  • Expose both BED chromosome conventions by category.
  • Expose ClinVar filtered resources by category.
  • Expose the assembly-level ClinVar database where required by variant confirmation.
  • Expose HPO, PharmCAT, RR-STR, and reference genome resources.
  • Ensure accessors are implemented as class methods or properties, not detached module-level functions.
  • Keep the underlying manifest available for provenance and debugging.

• Task 16 — Replace catalog generation with per-sample catalog selection

  • Replace the runtime catalog_generation step with a catalog-selection step.
  • Inspect chromosome naming independently for each sample VCF.
  • Select .bed or .chr.bed independently for each sample and category.
  • Select resources for the requested assembly and analysis category.
  • Register the selected BED and catalog JSON resources in the sample or execution context.
  • Do not use only the first sample VCF to select a shared BED.
  • Do not generate or modify persistent catalog files during sftool run.

  Suggested runtime entry point:

  run_catalog_selection(ctx)

• Task 17 — Integrate prepared ClinVar resources

  • Remove run_clinvar_setup() from sftool run.
  • Remove runtime calls to clinvar_manager() and any ClinVar download or transformation logic.
  • Use the Task 8b ClinVar JSON corresponding to:
    • execution assembly;
    • PR or RR catalog;
    • requested evidence level.
  • Adapt ClinVar consumers to obtain resources through ctx.resources.
  • Remove dependencies on legacy outputs such as:
    • ctx.outputs["clinvar"]["clinvar_db"];
    • ctx.outputs["clinvar"]["clinvar_summary_db"].
  • Define and implement the ClinVar resource used by variant_confirmation.
  • For variant-confirmation-only executions, use the assembly-level installed ClinVar database if ClinVar lookup is required, rather than forcing a PR/RR catalog selection.
  • Preserve ClinVar version and resource provenance for downstream reports where required.

• Task 18 — Adapt sample preprocessing

  • Use the reference genome resolved during bootstrap.
  • Normalize each sample VCF using ctx.resources.reference_genome.
  • Intersect normalized VCFs with the BED selected for that specific sample and category.
  • Stop reading catalog BED paths from legacy ctx.outputs["catalogs"]["bed_files"].
  • Use the installed PharmCAT positions VCF when required by the PharmCAT preprocessing workflow.
  • Preserve runtime executable configuration for bcftools, bgzip, Python, PharmCAT, and other tools.
  • Support different chromosome naming conventions across samples in the same execution.
  • Do not create or alter persistent shared resources.

• Task 19 — Adapt variant confirmation

  • Review the resource dependencies of run_variant_confirmation.
  • Use the normalized sample VCF generated during sample preprocessing.
  • Use the resolved reference genome.
  • Use the assembly-level installed ClinVar database when ClinVar annotation or lookup is required.
  • Avoid relying on PR/RR catalog resources unless the confirmation workflow explicitly needs them.
  • Support executions where variant_confirmation is the only enabled mode.
  • Preserve compatibility with combined secondary-findings and variant-confirmation executions.

• Task 20 — Adapt variant evidence preparation

  • Replace legacy catalog and ClinVar path access with ctx.resources.
  • Use the selected catalog JSON for each PR/RR category.
  • Use the selected catalog- and evidence-specific ClinVar JSON.
  • Continue supporting GeneBe-only and GeneBe-plus-ClinVar execution modes.
  • Avoid rebuilding ClinVar evidence collections from source databases at runtime.
  • Preserve the structures expected by downstream variant collection and selection.

• Task 21 — Adapt variant collection and supporting resources

  • Adapt run_variant_collection to consume installed resources.
  • Use the installed RR-STR CSV for STR-based RR analysis.
  • Use the installed PharmCAT positions resource where needed.
  • Use the installed HPO file for phenotype-based processing.
  • Remove legacy configuration-path dependencies for these datasets.
  • Preserve support for SNV/indel, STR, SMN1-copy, and PharmCAT collections.

• Task 22 — Adapt variant selection and report provenance

  • Review run_variant_selection and report-generation inputs for legacy resource paths.
  • Use resource versions and manifest metadata for provenance where appropriate.
  • Ensure selected variants retain the same expected data structures.
  • Do not expose installation-internal absolute paths unnecessarily in reports.
  • Preserve existing report contents unless resource provenance is intentionally added.

• Task 23 — Simplify run.py orchestration

  • Remove run_catalog_generation(ctx).
  • Remove run_clinvar_setup(ctx).
  • Add the catalog-selection stage.
  • Keep persistent resource validation and resolution in bootstrap.
  • Keep sample-specific selection and processing in runtime steps.
  • Remove obsolete resource-management conditionals from run.py.
  • Ensure report generation runs in normal executions as well as debug context reloads.
  • Keep the resulting orchestration focused on analysis stages.

  Expected high-level flow:

  ctx = bootstrap_execution(...)
  
  run_catalog_selection(ctx)
  run_sample_preprocessing(ctx)
  
  if "variant_confirmation" in ctx.modes:
      run_variant_confirmation(ctx)
  
  run_variant_evidence_preparation(ctx)
  run_variant_collection(ctx)
  run_variant_selection(ctx)
  run_report_generation(ctx)

  Individual stages may internally skip work when the relevant mode or category is not enabled.

Phase 6 — Cleanup, documentation, and validation

• Task 24 — Remove obsolete configuration fields

  • Remove installation-specific catalog, ClinVar, HPO, PharmCAT, RR-STR, and generated-resource paths that are now resolved from the manifest.
  • Retain runtime tool settings.
  • Retain supported reference genome overrides.
  • Update config validation accordingly.
  • Reject obsolete or unknown resource-path fields with clear errors where appropriate.

• Task 25 — Remove dead resource-generation code

  • Remove obsolete runtime catalog-generation code paths.
  • Remove obsolete runtime ClinVar download and transformation code paths.
  • Retain reusable transformation logic required by sftool resources setup.
  • Remove unused helpers, imports, context outputs, and configuration models.
  • Rename modules whose responsibility has changed, for example:
    • catalog_generation.py → catalog_selection.py;
    • remove clinvar_setup.py when it no longer has a runtime responsibility.

• Task 26 — Update documentation and examples

  • Document sftool resources setup.
  • Document the installed directory and manifest.
  • Document the two ways to provide a reference genome:
    • installed resource;
    • explicit runtime override.
  • Update example configuration files.
  • Update execution instructions.
  • Clarify that sftool run performs no persistent resource preparation.
  • Document supported ClinVar evidence levels and their cumulative behavior.

• Task 27 — Add unit and integration tests

  • Test manifest validation and checksum failures.
  • Test runtime resource resolution for both assemblies.
  • Test PR-only, RR-only, PR+RR, and no-catalog executions.
  • Test all supported ClinVar evidence levels.
  • Test configured reference overrides and installed references.
  • Test missing reference failures.
  • Test per-sample BED convention selection.
  • Test executions containing two samples with different chromosome conventions.
  • Test variant-confirmation-only execution.
  • Test combined secondary-findings and variant-confirmation execution.
  • Test GeneBe-only execution without unnecessary ClinVar consumption.

• Task 28 — Add end-to-end workflow tests

  • Test sftool resources setup.
  • Test repeated setup and idempotent resource reuse.
  • Test manifest generation and validation.
  • Test sftool run consuming installed resources.
  • Cover GRCh37 and GRCh38.
  • Cover PR and RR.
  • Cover BED and chr-BED VCF conventions.
  • Cover ClinVar evidence levels 1–4.
  • Verify that sftool run performs no download or persistent resource generation.
  • Verify report generation and resource provenance.

---

Runtime design notes

Resource preparation versus resource selection

Persistent resources are prepared only by:

sftool resources setup

Runtime stages may:

• validate;
• resolve;
• select;
• read;
• register selected paths in the execution context.

Runtime stages must not:

• download external datasets;
• query Ensembl to generate catalogs;
• generate shared catalog files;
• transform ClinVar source files;
• overwrite installed resources;
• rewrite the installed manifest.

Per-sample BED selection

BED selection must be sample-specific.

This is required because two VCFs in the same execution may use different chromosome conventions. The implementation must not infer the convention from only the first sample and then reuse that BED for all samples.

A possible structure is:

sample.resources["catalogs"][category] = {
    "bed": selected_bed_path,
    "json": catalog_json_path,
    "version": catalog_version,
}

The exact context structure may differ, provided that:

• the selected BED is unambiguous per sample and category;
• downstream preprocessing does not need to repeat chromosome-prefix detection;
• persistent installed descriptors remain available through ctx.resources.

ClinVar selection

For PR and RR analysis, bootstrap should select the prepared ClinVar JSON using:

assembly + category + evidence level

For variant confirmation, the workflow may require the assembly-level ClinVar database rather than a PR/RR-filtered resource. This distinction should be explicit in RuntimeResources.

Context responsibilities

Recommended separation:

• ctx.resources.installed

  • all validated installed resources;

• ctx.resources.execution

  • run-level resources selected from assembly, categories, evidence level, and configuration;

• sample.resources

  • sample-specific selections such as chromosome-convention-dependent BED files;

• ctx.outputs and sample.vcf_outputs

  • files produced by the current analysis execution.

Installed resource paths should not be treated as analysis outputs.

---

Acceptance criteria

• sftool resources setup installs all mandatory resources.
• PR and RR BED/JSON resources are generated for GRCh37 and GRCh38.
• Both chromosome naming conventions are available for every PR/RR BED.
• RR-STR remains a CSV catalog and does not produce BED or JSON resources.
• ClinVar source files and assembly-level databases are generated and retained.
• Sixteen ClinVar JSON resources are generated for PR/RR, GRCh37/GRCh38, and evidence levels 1–4.
• HPO and PharmCAT support resources are installed at stable paths.
• Reference genomes are downloaded only when requested.
• Every installed resource is represented in resources.json.
• Re-running setup reuses valid resources and produces a consistent manifest.
• Bootstrap validates installed resources before processing samples.
• Bootstrap resolves the selected assembly, categories, evidence level, reference genome, and shared datasets.
• BED convention selection is performed independently for each sample.
• Two samples with different chromosome naming conventions are supported in one execution.
• sftool run no longer generates catalog resources.
• sftool run no longer downloads or processes ClinVar source data.
• sftool run consumes only validated installed shared resources and runtime-generated sample outputs.
• Variant-confirmation-only execution has access to the required assembly-level resources.
• Existing GeneBe-only execution remains supported.
• Existing PR, RR, STR, SMN1-copy, PharmCAT, variant-selection, and report workflows remain functionally compatible.
• Obsolete configuration fields and dead runtime resource-generation code are removed.
• Documentation and end-to-end tests cover the complete resource installation and runtime consumption workflow.