Seqcol store app factory

README.md

@@ -7,10 +7,44 @@ User-facing documentation is hosted at [refgenie.org/refget](https://refgenie.or
 This repository includes:
 
 1. `/refget`: The `refget` Python package, which provides a Python interface to both remote and local use of refget standards. It has clients and functions for both refget sequences and refget sequence collections (seqcol).
-2. `/seqcolapi`: Sequence collections API software, a FastAPI wrapper built on top of the `refget` package. It provides a bare-bones Sequence Collections API service.
-3. `/deployment`: Server configurations for demo instances and public deployed instances. There are also github workflows (in `.github/workflows`) that deploy the demo server instance from this repository.
-4. `/test_fasta` and `/test_api`: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
-5. `/frontend`: a React seqcolapi front-end.
+2. `/refget/seqcolapi`: Sequence collections API software, a FastAPI wrapper built on top of the `refget` package. It provides a bare-bones Sequence Collections API service. It ships in the `refget` wheel, but its dependencies do not — see [Installation](#installation). Nothing on the plain `import refget` path imports this subpackage, so a base install never pays for fastapi/uvicorn/sqlmodel/psycopg2.
+3. `/seqcolapi`: A thin compatibility shim re-exporting `refget.seqcolapi`, kept so existing deployments that run `uvicorn seqcolapi.main:app` (or `:store_app`) keep working. **This shim is deliberately excluded from the wheel** — it is importable only from a checkout of this repository (or from a deployment that `COPY`s the directory), never from `pip install refget`. Every instruction below therefore uses `refget.seqcolapi.main`, which works everywhere.
+4. `/deployment`: Server configurations for demo instances and public deployed instances. There are also github workflows (in `.github/workflows`) that deploy the demo server instance from this repository.
+5. `/test_fasta` and `/test_api`: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
+6. `/frontend`: a React seqcolapi front-end.
+
+
+## Installation
+
+The base install is deliberately light — a client library and CLI, no web
+server and no ORM. Everything heavier is an extra, and the extras compose:
+
+| Install | Adds | Use it for |
+| --- | --- | --- |
+| `pip install refget` | — | The Python library and the `refget` CLI: digests, `RefgetStore`, API clients, compliance. No fastapi, no sqlalchemy. |
+| `pip install 'refget[db]'` | sqlmodel, psycopg2-binary | The SQLModel layer: `refget.models`, `refget.agents.RefgetDBAgent`, `refget admin`. A library capability — you can want the ORM without wanting a server. |
+| `pip install 'refget[seqcolapi]'` | fastapi, uvicorn | **Serving a RefgetStore.** `uvicorn refget.seqcolapi.main:store_app`, `refget store serve`, `refget.seqcolapi.create_seqcol_app`. No database of any kind. |
+| `pip install 'refget[seqcolapi-db]'` | both of the above | **The PostgreSQL-backed service** (`uvicorn refget.seqcolapi.main:app`), i.e. what runs seqcolapi.databio.org. |
+
+The importable module path is `refget.seqcolapi.main`, not `seqcolapi.main`. The
+bare `seqcolapi` package in this repository is a compatibility shim that is
+**not** shipped in the wheel; `uvicorn seqcolapi.main:store_app` resolves only
+from a checkout of this repository, and raises
+`ModuleNotFoundError: No module named 'seqcolapi'` in a pip-installed
+environment.
+
+The two service extras correspond to the two deployment modes described under
+[Development and deployment: Backend](#development-and-deployment-backend). The
+store-backed mode is the common case and the cheaper one; it needs no database
+dependencies at all.
+
+These boundaries are enforced by module structure, not convention: the
+database code lives in `refget/models.py`, `refget/agents.py` and
+`refget/seqcolapi/dbapp.py`, and nothing else imports them at module level. The
+router's response bodies live in `refget/response_models.py` (plain pydantic)
+precisely so that serving the API does not require an ORM. Importing a module
+without its extra raises an error naming the extra to install, rather than a
+bare `ModuleNotFoundError`. `tests/local/test_import_gating.py` is the tripwire.
 
 
 ## Deploy to AWS ECS
@@ -45,12 +79,15 @@ This starts the test database, runs tests, and cleans up automatically.
 
 ### Store-backed (no database)
 
-The store-backed seqcolapi uses a RefgetStore (local files) instead of PostgreSQL. This is the simplest way to run the API.
+The store-backed seqcolapi uses a RefgetStore (local files) instead of PostgreSQL. This is the simplest way to run the API, and it needs only `pip install 'refget[seqcolapi]'` — fastapi and uvicorn, no sqlmodel, no sqlalchemy, no psycopg2.
 
 For safe concurrent serving, the store is fully loaded and converted to a read-only store (`RefgetStore.into_readonly()`) before serving, so HTTP reads borrow immutably across request threads. The `refget store serve` CLI does this by default; pass `--lazy` to serve directly from the mutable, lazy-loading store instead (single-reader-oriented, not recommended for concurrent production serving).
 
 #### Quick start
 
+*Requires a checkout of this repository* (the script and the demo FASTA files
+live here):
+
 ```console
 bash deployment/store_demo_up.sh
 ```
@@ -64,29 +101,38 @@ No Docker or database required.
 
 #### Step-by-step
 
-1. Build a store from FASTA files:
+1. Build a store from FASTA files. `data_loaders/` is not part of the wheel, so
+   this step needs a checkout of this repository:
 
 ```console
 python data_loaders/demo_build_store.py test_fasta /tmp/refget_demo_store
 ```
 
-2. Start the store-backed API:
+   From a pip install, build a store with the CLI instead:
 
 ```console
-REFGET_STORE_PATH=/tmp/refget_demo_store uvicorn seqcolapi.main:store_app --reload --port 8100
+refget store init -p /tmp/refget_demo_store
+refget store add -p /tmp/refget_demo_store <your.fa>
+```
+
+2. Start the store-backed API. This works anywhere `refget[seqcolapi]` is
+   installed:
+
+```console
+REFGET_STORE_PATH=/tmp/refget_demo_store uvicorn refget.seqcolapi.main:store_app --reload --port 8100
 ```
 
 #### Remote store
 
 To run against a remote (S3) store:
 
 ```console
-REFGET_STORE_URL=https://example.com/store uvicorn seqcolapi.main:store_app --port 8100
+REFGET_STORE_URL=https://example.com/store uvicorn refget.seqcolapi.main:store_app --port 8100
 ```
 
 ### DB-backed (PostgreSQL)
 
-If you need a database-backed instance (e.g., for mutable data, advanced queries), use the DB-backed workflow. In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way to get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):
+If you need a database-backed instance (e.g., for mutable data, advanced queries), use the DB-backed workflow. This one needs `pip install 'refget[seqcolapi-db]'`. In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way to get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):
 
 ```console
 bash deployment/demo_up.sh
@@ -129,23 +175,25 @@ If you need to load test data into your server, then you have to install [gtars]
 PYTHONPATH=. python data_loaders/load_demo_seqcols.py
 ```
 
-or:
+or, with the CLI (`refget add-fasta` was replaced by `refget admin load`):
 
 ```
-refget add-fasta -p test_fasta/test_fasta_metadata.csv -r test_fasta
+refget admin load --pep test_fasta/test_fasta_metadata.csv --fa-root test_fasta
 ```
 
 #### Running the seqcolapi API backend
 
 Run the demo `seqcolapi` service like this:
 
 ```
-uvicorn seqcolapi.main:app --reload --port 8100
+uvicorn refget.seqcolapi.main:app --reload --port 8100
 ```
 
 #### Running with docker
 
-To build the docker file, first build the image from the root of this repository:
+To build the docker file, first build the image from a checkout of this
+repository (the build context is the `seqcolapi/` compatibility shim, which
+exists only here):
 
 ```
 docker build -f deployment/dockerhub/Dockerfile -t databio/seqcolapi seqcolapi
@@ -243,13 +291,21 @@ interface SeqColResult {
 
 ### Models
 
-The objects and attributes are represented as SQLModel objects in `refget/models.py`. To add a new attribute:
+The database objects and attributes are represented as SQLModel objects in `refget/models.py` (requires `refget[db]`). To add a new attribute:
 
 1. create a new model. This will create a table for that model, etc.
 2. change the function that creates the objects, to populate the new attribute.
 
+HTTP response bodies are *not* defined there. They are plain pydantic models in
+`refget/response_models.py`, so that `refget.router` — and therefore the
+store-backed service — can be imported without an ORM. Put a new response
+schema there unless it genuinely maps to a database table.
+
 ## Example of loading reference fasta datasets:
 
+Needs `pip install 'refget[seqcolapi-db]'` and a configured PostgreSQL
+connection (see [DB-backed (PostgreSQL)](#db-backed-postgresql)):
+
 ```
-refget add-fasta -p ref_fasta.csv -r $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta
+refget admin load --pep ref_fasta.csv --fa-root $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta
 ```

deployment/demo_up.sh

@@ -35,7 +35,7 @@ docker run --rm -d --name refget-postgres -p 127.0.0.1:5432:5432 \
   echo "Postgres is up - continuing"
 
 echo "Running uvicorn API service..."
-uvicorn seqcolapi.main:app --reload --port 8100 &
+uvicorn refget.seqcolapi.main:app --reload --port 8100 &
 PID=$!
 
 echo "Loading demo sequence collections..."

deployment/seqcolapi-store/Dockerfile

@@ -1,11 +1,15 @@
 FROM tiangolo/uvicorn-gunicorn:python3.11-slim
 LABEL authors="Nathan Sheffield"
-RUN pip install https://github.com/refgenie/refget/archive/dev.zip
-RUN pip install gtars
 
-COPY seqcolapi/requirements/requirements-seqcolapi.txt requirements-seqcolapi.txt
-RUN pip install -r requirements-seqcolapi.txt --no-cache-dir
+# The store-backed service needs no database packages. `refget[seqcolapi]` pulls
+# fastapi and uvicorn only, so sqlmodel, sqlalchemy and psycopg2 stay out of the
+# image. The Postgres-backed service is a separate deployment and installs
+# `refget[seqcolapi-db]`.
+#
+# seqcolapi now ships inside the refget wheel (refget/seqcolapi/), so there is
+# nothing to COPY -- the app is importable straight from site-packages. gtars is
+# a base dependency of refget and no longer needs its own install line.
+RUN pip install --no-cache-dir \
+    "refget[seqcolapi] @ https://github.com/refgenie/refget/archive/dev.zip"
 
-COPY seqcolapi/ /app/seqcolapi
-
-CMD ["uvicorn", "seqcolapi.main:store_app", "--host", "0.0.0.0", "--port", "80"]
+CMD ["uvicorn", "refget.seqcolapi.main:store_app", "--host", "0.0.0.0", "--port", "80"]

deployment/store_demo_up.sh

@@ -44,7 +44,7 @@ STORE_HTTP_PID=$!
 export REFGET_STORE_HTTP_URL="http://localhost:$STORE_HTTP_PORT"
 
 echo "Running store-backed uvicorn API service..."
-uvicorn seqcolapi.main:store_app --reload --port ${SEQCOLAPI_PORT:-8100} &
+uvicorn refget.seqcolapi.main:store_app --reload --port ${SEQCOLAPI_PORT:-8100} &
 PID=$!
 
 echo ""

mkdocs.yml

@@ -10,7 +10,9 @@ nav:
     - "Tutorial": tutorial.md
   - Reference:
     - Support: https://github.com/refgenie/refget/issues
-    - Changelog: changelog.md
+    # The changelog lives in the refgenie-docs repo, at
+    # src/content/docs/refget/reference/changelog.md
+    - Changelog: https://refgenie.org/refget/reference/changelog/
 
 theme: databio
 

models_notes.md

@@ -49,7 +49,6 @@ Digest is a string, and value is a JSON column, where I put the content of the a
 This is some old deprecated models I had been working on, under different modeling approaches; probably can delete these soon if I don't need to revisit it.
 
 ```python
-
 # DigestedSequenceCollection = create_model(
 # 	'DigestedSequenceCollection', digest=(str, ""),
 # 	__base__= SequenceCollection,
@@ -134,7 +133,6 @@ This is some old deprecated models I had been working on, under different modeli
 
 
 if False:
-
     from pydantic import create_model
     from typing import List
     from sqlmodel import Field, ARRAY, SQLModel, create_engine, Column, Float, Relationship

pyproject.toml

@@ -23,12 +23,16 @@ classifiers = [
     "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: 3.14",
 ]
+# Base install must stay light: no fastapi, no uvicorn, no sqlmodel, no
+# psycopg2. Those are service/database dependencies, gated behind the
+# `seqcolapi` / `db` extras and behind the refget.seqcolapi / refget.agents /
+# refget.models module boundaries -- nothing on the `import refget` path
+# touches them.
 dependencies = [
     "gtars>=0.9.0",
     "jsonschema",
     "pyyaml",
     "requests",
-    "sqlmodel",
     "tomli_w",
     "typer>=0.9.0",
 ]
@@ -37,21 +41,54 @@ dependencies = [
 refget = "refget.cli:main"
 
 [project.optional-dependencies]
-test = ["pytest", "pytest-cov>=6.0.0", "fastapi", "httpx"]
-seqcolapi = [
-    "fastapi",
-    "psycopg2-binary",
-    "sqlmodel",
-    "uvicorn>=0.30.0",
-    "ubiquerg>=0.6.1",
-]
+test = ["pytest", "pytest-cov>=6.0.0", "fastapi", "httpx", "sqlmodel"]
+
+# There are three axes here, and they compose:
+#
+#   db            the SQLModel/SQLAlchemy layer -- refget.models, refget.agents,
+#                 `refget admin`. Its own extra, not a corner of `seqcolapi`,
+#                 because it is a *library* capability: you can want the ORM
+#                 without wanting a web server.
+#   seqcolapi     the web service -- fastapi + uvicorn, nothing more. This is
+#                 all the RefgetStore-backed deployment needs. No ORM: nothing
+#                 reachable from refget.seqcolapi.app imports sqlmodel (the
+#                 router's response models live in refget/response_models.py,
+#                 and setup_backend imports RefgetDBAgent only in its `engine`
+#                 branch).
+#   seqcolapi-db  the PostgreSQL-backed deployment (seqcolapi.databio.org):
+#                 both of the above. Named rather than left for the user to
+#                 spell as `refget[seqcolapi,db]`, because the two service
+#                 modes are a real deployment choice and should be
+#                 discoverable from the extras list.
+#
+# `seqcolapi` keeps its name and its meaning-for-most-people (run the service),
+# so nobody's existing install line breaks; it just stopped dragging in an ORM
+# that the store-backed app never touches.
+#
+# ubiquerg was listed here and is imported by nothing under refget/ -- dropped.
+db = ["psycopg2-binary", "sqlmodel"]
+seqcolapi = ["fastapi", "uvicorn>=0.30.0"]
+seqcolapi-db = ["refget[db,seqcolapi]"]
 
 [project.urls]
 Homepage = "https://github.com/refgenie/refget"
 
 [tool.hatch.version]
 path = "refget/_version.py"
 
+# Ship exactly one package: refget/. Stated explicitly rather than left to
+# hatchling's name-based auto-detection, because there is now a second
+# importable top-level directory -- the seqcolapi/ compatibility shim -- which
+# must NOT go in the wheel. The shim exists only for deployments that COPY it
+# out of the repo; the real code is refget/seqcolapi/.
+#
+# refget/seqcolapi/static/ needs no force-include: it lives inside the package
+# directory, and hatchling ships every non-VCS-ignored file it finds there.
+# Verify after touching .gitignore -- an ignore rule that matched, say, *.ico
+# would silently drop package data from the wheel.
+[tool.hatch.build.targets.wheel]
+packages = ["refget"]
+
 # Bundle the compliance known-answer fixtures into the installed package so the
 # compliance runner is self-contained on a pip-installed deploy (the source
 # files live at the repo root, outside the refget/ package).
@@ -72,6 +109,15 @@ exclude = [
 select = ["E", "F", "I"]
 ignore = ["F403", "F405", "E501"]
 
+[tool.ruff.lint.per-file-ignores]
+# The dependency gate has to run before the imports it guards, so those imports
+# are deliberately not at the top of the file.
+"refget/seqcolapi/__init__.py" = ["E402"]
+"refget/seqcolapi/dbapp.py" = ["E402"]
+"refget/models.py" = ["E402"]
+"refget/agents.py" = ["E402"]
+"refget/router.py" = ["E402"]
+
 [tool.ruff.lint.isort]
 known-first-party = ["refget"]
 

refget/__init__.py

@@ -7,7 +7,14 @@
     from refget.utils import compare_seqcols, validate_seqcol, seqcol_digest
     from refget.clients import SequenceCollectionClient, FastaDrsClient
     from refget.router import create_refget_router
+    from refget.seqcolapi import create_seqcol_app, prepare_store
     from refget.agents import RefgetDBAgent
+
+`refget.seqcolapi` (the API service) and `refget.router` require the optional
+`fastapi` dependency; `refget.agents` and `refget.models` require `sqlmodel`.
+None of them are imported here, and nothing on this base-install path imports
+them, so `pip install refget` never pays for them. Install the service with
+`pip install 'refget[seqcolapi]'`.
 """
 
 from ._version import __version__

refget/_deps.py

@@ -0,0 +1,52 @@
+"""Optional-dependency gates.
+
+refget ships several modules that require optional dependencies: the service
+subpackage needs fastapi, the database modules need sqlmodel/sqlalchemy. Those
+dependencies are imported at module level inside the modules that own them --
+see the module-boundary rule in ``refget/models.py`` and
+``refget/seqcolapi/__init__.py``. This module supplies the one thing that has
+to run *before* those imports: a check that turns a missing extra into a
+sentence telling you what to type, instead of a ``ModuleNotFoundError`` raised
+from four imports deep in somebody else's package.
+
+Keep this module import-light. It is loaded on paths that must not pull in the
+very dependencies it is checking for, so it probes with
+``importlib.util.find_spec`` rather than importing.
+"""
+
+from importlib.util import find_spec
+
+__all__ = ["require"]
+
+
+def _findable(name: str) -> bool:
+    try:
+        return find_spec(name) is not None
+    except (ImportError, ValueError):
+        # ImportError: a parent package is itself missing or broken.
+        # ValueError: the module is present in sys.modules but has no spec.
+        return False
+
+
+def require(what: str, extra: str, *modules: str) -> None:
+    """Raise an actionable ImportError if any of ``modules`` is unavailable.
+
+    Probes rather than wrapping the caller's imports, so that a genuine
+    ImportError raised from *inside* an installed dependency still surfaces as
+    itself instead of being mistranslated into "not installed".
+
+    Args:
+        what: What the caller is, phrased for the error message, e.g.
+            ``"refget.models defines the SQLModel database tables and"``.
+        extra: The pip extra that supplies the dependencies, e.g. ``"db"``.
+            Rendered as ``pip install 'refget[db]'``.
+        *modules: Top-level module names to probe.
+    """
+    missing = [name for name in modules if not _findable(name)]
+    if not missing:
+        return
+    raise ImportError(
+        f"{what} requires the optional '{extra}' dependencies, and "
+        f"{', '.join(missing)} {'is' if len(missing) == 1 else 'are'} not "
+        f"installed.\nInstall them with:  pip install 'refget[{extra}]'"
+    )