diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index b112fd0f2..1bd58ad19 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -28,6 +28,7 @@ * [gget mutate](en/mutate.md) * [gget opentargets](en/opentargets.md) * [gget pdb](en/pdb.md) +* [gget pineapple](en/pineapple.md) * [gget ref](en/ref.md) * [gget search](en/search.md) * [gget setup](en/setup.md) @@ -71,6 +72,7 @@ * [gget mutate](es/mutate.md) * [gget opentargets](es/opentargets.md) * [gget pdb](es/pdb.md) +* [gget pineapple](es/pineapple.md) * [gget ref](es/ref.md) * [gget search](es/search.md) * [gget setup](es/setup.md) diff --git a/docs/src/en/pineapple.md b/docs/src/en/pineapple.md new file mode 100644 index 000000000..fa8b2b22c --- /dev/null +++ b/docs/src/en/pineapple.md @@ -0,0 +1,83 @@ +[ View page source on GitHub ](https://github.com/scverse/gget/blob/main/docs/src/en/pineapple.md) + +> Python arguments are equivalent to long-option arguments (`--arg`), unless otherwise specified. Flags are True/False arguments in Python. The manual for any gget tool can be called from the command-line using the `-h` `--help` flag. +# gget pineapple 🍍 +List and download curated bio-imaging datasets and pre-trained model weights from [Pineapple](https://github.com/tomouellette/pineapple). +[Pineapple](https://github.com/tomouellette/pineapple) (by Tom Ouellette) is a tool for image-based cell profiling that also curates and standardizes a collection of annotated bio-imaging datasets (segmentation and benchmark) and self-supervised model weights, hosted on Google Drive. `gget pineapple` lets you browse this catalog and download the resources directly — no Rust binary required. +Return format: JSON (command-line) or data frame/CSV (Python). + +> **Scope:** `gget pineapple` wraps only Pineapple's data download. It does not reimplement Pineapple's image-processing features (`process`, `profile`, `neural`, `measure`) — use the [Pineapple](https://github.com/tomouellette/pineapple) tool directly for those. + +> ⚠️ Please check each dataset's original reference and license (shown in the catalog) before use. Some datasets are non-commercial only. + +**Positional argument** +`name` +Name of the dataset/weights to fetch, e.g. `vicar_2021` or `dino_vit_small`. Omit to list the full catalog for the chosen category. + +**Optional arguments** +`-c` `--category` +Resource category: `segmentation`, `benchmark`, or `weights`. Default: `segmentation`. + +`-od` `--out_dir` +Directory to download the resource into (used with `--download`). Default: current directory. + +`-o` `--out` +Path to the file the catalog table will be saved in, e.g. path/to/directory/results.csv (or .json). Default: Standard out. +Python: `save=True` will save the output in the current working directory. + +**Flags** +`-d` `--download` +Download the resource (requires a specific `name`) into `--out_dir`. Datasets are large (up to several GB); see the `size_gb` column. + +`-csv` `--csv` +Command-line only. Returns results in CSV format. +Python: Use `json=True` to return output in JSON format. + +`-q` `--quiet` +Command-line only. Prevents progress information from being displayed. +Python: Use `verbose=False` to prevent progress information from being displayed. + +### Examples +**List the available segmentation datasets:** +```bash +gget pineapple --category segmentation +``` +```python +# Python +gget.pineapple(category="segmentation") +``` +→ Returns the catalog of curated segmentation datasets. + +| name | category | data_authors | size_gb | license | filename | google_drive_id | +| --- | --- | --- | --- | --- | --- | --- | +| vicar_2021 | segmentation | Vicar et al. 2021 | 0.113 | CC BY 4.0 | vicar-2021.tar.gz | 12tJOlIHZPFqp8GLek_jV__Uhhgsa530_ | +| . . . | . . . | . . . | . . . | . . . | . . . | . . . | + +

+**Download a specific dataset:** +```bash +gget pineapple vicar_2021 --download --out_dir ./pineapple_data +``` +```python +# Python +gget.pineapple("vicar_2021", download=True, out_dir="./pineapple_data") +``` +→ Downloads `vicar-2021.tar.gz` into `./pineapple_data` and returns the catalog entry. + +

+**List the pre-trained model weights:** +```bash +gget pineapple --category weights +``` +```python +# Python +gget.pineapple(category="weights") +``` +→ Returns the catalog of pre-trained self-supervised model weights (e.g. `dino_vit_small`, `subcell_vit_base`). + +# References +If you use `gget pineapple` in a publication, please cite the following article and the original dataset references (listed in the catalog `data_authors` column): + +- Pineapple: scalable processing for image-based cell profiling. [https://github.com/tomouellette/pineapple](https://github.com/tomouellette/pineapple) + +- Luebbert, L., & Pachter, L. (2023). Efficient querying of genomic reference databases with gget. Bioinformatics. [https://doi.org/10.1093/bioinformatics/btac836](https://doi.org/10.1093/bioinformatics/btac836) diff --git a/docs/src/en/updates.md b/docs/src/en/updates.md index 89be22281..385898d0f 100644 --- a/docs/src/en/updates.md +++ b/docs/src/en/updates.md @@ -5,6 +5,7 @@ #### *gget* officially became part of [*scverse*](https://scverse.org/) on June 9, 2026. 🥳🥳🥳 **Version ≥ 0.30.9** (XXX XX, 2026): +- [`gget pineapple`](pineapple.md): **New module** to list and download curated bio-imaging datasets and pre-trained model weights from [Pineapple](https://github.com/tomouellette/pineapple) (by Tom Ouellette). Browse the segmentation/benchmark dataset and model-weights catalog (names, authors, sizes, licenses) and download a resource by name directly from its Google Drive host — no Rust binary required. Available in the Python API and on the command line. Resolves [issue 161](https://github.com/scverse/gget/issues/161). **Version ≥ 0.30.8** (Jun 28, 2026): - [`gget g2p`](g2p.md): Either `gene` or `--uniprot_id` is now sufficient — whichever is missing is resolved via UniProt and cached. Gene→UniProt picks the canonical reviewed human Swiss-Prot entry; the resolution and its limitations are logged. The canonical pair is **always** prepended to the result as `gene_name` / `uniprot_id` columns (and stored on `df.attrs`), so the output schema is invariant regardless of input mode. Existing call sites continue to work. diff --git a/docs/src/es/pineapple.md b/docs/src/es/pineapple.md new file mode 100644 index 000000000..d24962b08 --- /dev/null +++ b/docs/src/es/pineapple.md @@ -0,0 +1,83 @@ +[ Ver el codigo fuente de la pagina en GitHub ](https://github.com/scverse/gget/blob/main/docs/src/es/pineapple.md) + +> Parámetros de Python són iguales a los parámetros largos (`--parámetro`) de Terminal, si no especificado de otra manera. Banderas son parámetros de verdadero o falso (True/False) en Python. El manuál para cualquier modulo de gget se puede llamar desde la Terminal con la bandera `-h` `--help`. +# gget pineapple 🍍 +Lista y descarga conjuntos de datos de bioimagen curados y pesos de modelos preentrenados desde [Pineapple](https://github.com/tomouellette/pineapple). +[Pineapple](https://github.com/tomouellette/pineapple) (por Tom Ouellette) es una herramienta para el perfilado celular basado en imágenes que además cura y estandariza una colección de conjuntos de datos de bioimagen anotados (segmentación y benchmark) y pesos de modelos autosupervisados, alojados en Google Drive. `gget pineapple` te permite explorar este catálogo y descargar los recursos directamente — sin necesidad del binario de Rust. +Regresa: JSON (línea de comandos) o Dataframe/CSV (Python). + +> **Alcance:** `gget pineapple` solo envuelve la descarga de datos de Pineapple. No reimplementa las funciones de procesamiento de imágenes de Pineapple (`process`, `profile`, `neural`, `measure`) — para eso, usa directamente la herramienta [Pineapple](https://github.com/tomouellette/pineapple). + +> ⚠️ Por favor revisa la referencia original y la licencia de cada conjunto de datos (mostradas en el catálogo) antes de usarlo. Algunos conjuntos de datos son solo para uso no comercial. + +**Parámetro posicional** +`name` +Nombre del conjunto de datos/pesos a obtener, p. ej. `vicar_2021` o `dino_vit_small`. Omítelo para listar el catálogo completo de la categoría elegida. + +**Parámetros optionales** +`-c` `--category` +Categoría del recurso: `segmentation`, `benchmark`, o `weights`. Por defecto: `segmentation`. + +`-od` `--out_dir` +Directorio en el que se descargará el recurso (se usa con `--download`). Por defecto: directorio actual. + +`-o` `--out` +Ruta del archivo en el que se guardará la tabla del catálogo, p. ej. ruta/al/directorio/results.csv (o .json). Por defecto: salida estándar. +Python: `save=True` guardará la salida en el directorio de trabajo actual. + +**Banderas** +`-d` `--download` +Descarga el recurso (requiere un `name` específico) en `--out_dir`. Los conjuntos de datos son grandes (hasta varios GB); consulta la columna `size_gb`. + +`-csv` `--csv` +Solo para Terminal. Produce los resultados en formato CSV. +Para Python, usa `json=True` para producir la salida en formato JSON. + +`-q` `--quiet` +Solo para Terminal. Impide que la información de progreso sea exhibida. +Para Python, usa `verbose=False` para impedir que la información de progreso sea exhibida. + +### Ejemplos +**Listar los conjuntos de datos de segmentación disponibles:** +```bash +gget pineapple --category segmentation +``` +```python +# Python +gget.pineapple(category="segmentation") +``` +→ Produce el catálogo de conjuntos de datos de segmentación curados. + +| name | category | data_authors | size_gb | license | filename | google_drive_id | +| --- | --- | --- | --- | --- | --- | --- | +| vicar_2021 | segmentation | Vicar et al. 2021 | 0.113 | CC BY 4.0 | vicar-2021.tar.gz | 12tJOlIHZPFqp8GLek_jV__Uhhgsa530_ | +| . . . | . . . | . . . | . . . | . . . | . . . | . . . | + +

+**Descargar un conjunto de datos específico:** +```bash +gget pineapple vicar_2021 --download --out_dir ./pineapple_data +``` +```python +# Python +gget.pineapple("vicar_2021", download=True, out_dir="./pineapple_data") +``` +→ Descarga `vicar-2021.tar.gz` en `./pineapple_data` y produce la entrada del catálogo. + +

+**Listar los pesos de modelos preentrenados:** +```bash +gget pineapple --category weights +``` +```python +# Python +gget.pineapple(category="weights") +``` +→ Produce el catálogo de pesos de modelos autosupervisados preentrenados (p. ej. `dino_vit_small`, `subcell_vit_base`). + +# Citar +Si utiliza `gget pineapple` en una publicación, favor de citar el siguiente recurso y las referencias originales de los conjuntos de datos (listadas en la columna `data_authors` del catálogo): + +- Pineapple: scalable processing for image-based cell profiling. [https://github.com/tomouellette/pineapple](https://github.com/tomouellette/pineapple) + +- Luebbert, L., & Pachter, L. (2023). Efficient querying of genomic reference databases with gget. Bioinformatics. [https://doi.org/10.1093/bioinformatics/btac836](https://doi.org/10.1093/bioinformatics/btac836) diff --git a/gget/__init__.py b/gget/__init__.py index f56cbcddc..860ce3f4e 100644 --- a/gget/__init__.py +++ b/gget/__init__.py @@ -22,6 +22,7 @@ from .gget_mutate import mutate from .gget_opentargets import opentargets from .gget_pdb import pdb +from .gget_pineapple import pineapple from .gget_ref import ref from .gget_search import search from .gget_seq import seq diff --git a/gget/constants.py b/gget/constants.py index 38f90119f..6c4f59ac5 100644 --- a/gget/constants.py +++ b/gget/constants.py @@ -7,6 +7,10 @@ # strategy avoid hanging indefinitely on slow upstreams. DEFAULT_REQUESTS_TIMEOUT = (10, 60) +# Google Drive download endpoint used by gget pineapple +# (the Pineapple bio-imaging datasets/weights are hosted on Google Drive) +PINEAPPLE_GDRIVE_URL = "https://drive.google.com/uc?export=download" + # Ensembl REST API server for gget seq and info ENSEMBL_REST_API = "http://rest.ensembl.org/" ENSEMBL_FTP_URL = "http://ftp.ensembl.org/pub/" diff --git a/gget/gget_pineapple.py b/gget/gget_pineapple.py new file mode 100644 index 000000000..e3aecd0b8 --- /dev/null +++ b/gget/gget_pineapple.py @@ -0,0 +1,432 @@ +from __future__ import annotations + +import json as json_package +import os +from typing import Any, Literal, overload + +import pandas as pd +import requests +from bs4 import BeautifulSoup, Tag + +from .constants import DEFAULT_REQUESTS_TIMEOUT, PINEAPPLE_GDRIVE_URL +from .utils import set_up_logger + +logger = set_up_logger() + +# --------------------------------------------------------------------------- +# Pineapple data catalog. +# +# Pineapple (https://github.com/tomouellette/pineapple) is a command-line tool +# for processing/profiling morphological data in bio-imaging datasets. Its +# `pineapple download` command distributes a curated set of standardized +# bio-imaging datasets and pre-trained model weights, each hosted on Google +# Drive. The file IDs, licenses, authors, and sizes below are mirrored verbatim +# from the pineapple `pineapple-data` crate so that gget can list and download +# the same resources without requiring the Rust binary. +# --------------------------------------------------------------------------- + +_SEGMENTATION: dict[str, dict[str, str]] = { + "almeida_2023": { + "file_id": "1BlHvG0MkWwuqGA3ImUJ09D9E7rsVt5ER", + "data_authors": "Almeida et al. 2023", + "size_gb": "0.927", + "license": "CC BY 4.0", + }, + "arvidsson_2022": { + "file_id": "12Cwk5MX3V9z_2KmBJyn5jXc-JuW7-e2k", + "data_authors": "Arvidsson et al. 2022", + "size_gb": "0.028", + "license": "CC BY 4.0", + }, + "cellpose_2021": { + "file_id": "12Z9PpJEdSE0bHALNxpAAeD6WA9aBhMEO", + "data_authors": "Stringer et al. 2021", + "size_gb": "0.356", + "license": "Custom NC", + }, + "conic_2022": { + "file_id": "1nXOnDkWpRfU5iGXFZe06-CQaAMFq13f_", + "data_authors": "Graham et al. 2022", + "size_gb": "1.920", + "license": "CC BY-NC 4.0", + }, + "cryonuseg_2021": { + "file_id": "1cfIY9BSlTe0RNaq1V8fZmKJwWyBs4WEj", + "data_authors": "Mahbod et al. 2021", + "size_gb": "0.031", + "license": "MIT", + }, + "dsb_2019": { + "file_id": "1qgAyMcrZwLudlA4vjy7jwuKTjAxT7Ky2", + "data_authors": "Caicedo et al. 2019", + "size_gb": "0.112", + "license": "CC0 1.0 Universal", + }, + "hpa_2022": { + "file_id": "1NyV6xuIAIuaSiXp0H-4VCV8tjaNSpXtX", + "data_authors": "HPA 2022", + "size_gb": "1.630", + "license": "CC BY 4.0", + }, + "livecell_2021": { + "file_id": "1JNXkZS0QSQW25b-opoyKomPKCfO_3pkx", + "data_authors": "Edlund et al. 2021", + "size_gb": "3.260", + "license": "CC BY-NC 4.0", + }, + "nuinseg_2024": { + "file_id": "1gSmbsfhO7aP1yBB5R9XMMrAH4hy-Thmm", + "data_authors": "Mahbod et al. 2024", + "size_gb": "0.347", + "license": "MIT", + }, + "pannuke_2020": { + "file_id": "1J9CeH9t23EpottNyUKeBBkYpTfMR3EgT", + "data_authors": "Gamper et al. 2020", + "size_gb": "1.250", + "license": "CC BY-NC-SA 4.0", + }, + "tissuenet_2022": { + "file_id": "1ilHrzUuGfobSdFmTezyynCWCLIoJwaHQ", + "data_authors": "Greenwald et al. 2022", + "size_gb": "4.270", + "license": "Modified NC Apache", + }, + "vicar_2021": { + "file_id": "12tJOlIHZPFqp8GLek_jV__Uhhgsa530_", + "data_authors": "Vicar et al. 2021", + "size_gb": "0.113", + "license": "CC BY 4.0", + }, +} + +_BENCHMARK: dict[str, dict[str, str]] = { + "amgad_2022": { + "file_id": "1JHlGon82bYPhpeOwbRYxz4uRxhcxasr3", + "data_authors": "Amgad et al. 2022", + "size_gb": "0.062", + "license": "CC0 1.0", + }, + "cnmc_2019": { + "file_id": "1a7Wt0kwt3Uq1NKMtBsWmesMH4tCwqkgi", + "data_authors": "C-NMC Challenge", + "size_gb": "0.182", + "license": "CC BY 3.0", + }, + "fracatlas_2023": { + "file_id": "1vyXNA4bxMFk-7Hw59TPiWIfX-BzTSmCd", + "data_authors": "Abedeen et al. 2023", + "size_gb": "0.247", + "license": "CC BY 4.0", + }, + "isic_2019": { + "file_id": "1CDGbcBxs7SUemGwpBtoVYJblqCNgw469", + "data_authors": "ISIC", + "size_gb": "1.140", + "license": "CC BY-NC 4.0", + }, + "kermany_2018": { + "file_id": "1Xk7LWa7HWzTN7Nxsa8MuefBmzNsz4VuM", + "data_authors": "Kermany et al. 2018", + "size_gb": "0.638", + "license": "CC BY 4.0", + }, + "kromp_2023": { + "file_id": "16RXNWQXlw_scJ75DwowngxJHYB2rZsW7", + "data_authors": "Kromp et al. 2023", + "size_gb": "0.025", + "license": "CC BY 4.0", + }, + "matek_2021": { + "file_id": "1BDYtZoqSUUZQmWgcEBtopaJTWgwrAXGz", + "data_authors": "Matek et al. 2021", + "size_gb": "0.508", + "license": "CC BY 4.0", + }, + "murphy_2001": { + "file_id": "1fl4dwbjX11SpDRhwbIi2-lbcswvyr1F_", + "data_authors": "Murphy et al. 2001", + "size_gb": "0.033", + "license": "MIT", + }, + "opencell_2024": { + "file_id": "1nlqt7ujUPciEoAKriIu_bqE5fUJZX4nx", + "data_authors": "OpenCell", + "size_gb": "1.030", + "license": "MIT", + }, + "phillip_2021": { + "file_id": "1yE4BblXBAPJDT1AnK3gghHAS3cZUFCd6", + "data_authors": "Phillip et al. 2021", + "size_gb": "0.032", + "license": "MIT", + }, + "recursion_2019": { + "file_id": "1209hlaKcOqKdEGOwvlRhJakX8ciN-SX8", + "data_authors": "Recursion", + "size_gb": "0.037", + "license": "CC BY-NC-SA 4.0", + }, + "verma_2021": { + "file_id": "1AyU-4-doJY2GX3dmf7ryPDCvDA_x4PPD", + "data_authors": "Verma et al. 2021", + "size_gb": "0.021", + "license": "CC BY-NC-SA 4.0", + }, + "runtime": { + "file_id": "1BlXIv49oxj2dsiiEASbTEiyh7QpIjYb_", + "data_authors": "MIT", + "size_gb": "0.017", + "license": "MIT", + }, +} + +_WEIGHTS: dict[str, dict[str, str]] = { + "dino_vit_small": { + "file_id": "1xuyTyPsuPiDtec8ojZwAyXSDq9AzyPQX", + "filename": "dinov2_vits14_imagenet.safetensors", + "data_authors": "Huggingface/candle", + "size_gb": "0.097", + "license": "Apache License 2.0", + }, + "dino_vit_base": { + "file_id": "19vy-A-KTaaF3vsWKxu0JpA0gaATU52Gh", + "filename": "dinov2_vitb14_imagenet.safetensors", + "data_authors": "Huggingface/candle", + "size_gb": "0.330", + "license": "Apache License 2.0", + }, + "dinobloom_vit_base": { + "file_id": "1XhzSiO2IDKppr2UCTAio_niLSk5QA6hG", + "filename": "dinov2_vitb14_dinobloom.safetensors", + "data_authors": "Marr Lab", + "size_gb": "0.330", + "license": "Apache License 2.0", + }, + "scdino_vit_small": { + "file_id": "1omwQNJVMkrbYCstSF11p5HsErHzINTz6", + "filename": "scdino_vit_small.safetensors", + "data_authors": "Snijder Lab", + "size_gb": "0.097", + "license": "Apache License 2.0", + }, + "subcell_vit_base": { + "file_id": "1LZn3xlgVVd2jQIpXst4CMCYN58F-VG-x", + "filename": "subcell_vit_base.safetensors", + "data_authors": "Lundberg Lab", + "size_gb": "0.330", + "license": "MIT License", + }, +} + +_CATEGORIES: dict[str, dict[str, dict[str, str]]] = { + "segmentation": _SEGMENTATION, + "benchmark": _BENCHMARK, + "weights": _WEIGHTS, +} + +# Closed set of resource categories. Used as the public `category` type so editors +# and type checkers can flag typos; the implementation still accepts a plain str +# (it lowercases and validates at runtime, listing valid values on error). +_CategoryT = Literal["segmentation", "benchmark", "weights"] + +_COLUMNS = ["name", "category", "data_authors", "size_gb", "license", "filename", "google_drive_id"] + + +def _resource_filename(category: str, name: str, info: dict[str, str]) -> str: + """Return the filename pineapple uses for a dataset/weights resource.""" + if category == "weights": + return info["filename"] + # Datasets are distributed as tarballs with dashes instead of underscores + return f"{name.replace('_', '-')}.tar.gz" + + +def _catalog_row(category: str, name: str, info: dict[str, str]) -> dict[str, Any]: + """Build a catalog row for a single resource.""" + return { + "name": name, + "category": category, + "data_authors": info.get("data_authors"), + "size_gb": float(info["size_gb"]) if info.get("size_gb") else None, + "license": info.get("license"), + "filename": _resource_filename(category, name, info), + "google_drive_id": info.get("file_id"), + } + + +def _parse_gdrive_form(html_text: str) -> tuple[str | None, dict[str, str]]: + """Parse Google Drive's virus-scan-warning form (for large files). + + Mirrors the behavior of pineapple's Rust downloader: locate the + `form#download-form` and collect its named inputs to build the confirmed + download request. + """ + soup = BeautifulSoup(html_text, "html.parser") + form = soup.find("form", id="download-form") + if not isinstance(form, Tag): + return None, {} + action = form.get("action") + if isinstance(action, list): # a multi-valued attr; the download form's is single + action = action[0] if action else None + params: dict[str, str] = {} + for inp in form.find_all("input"): + if not isinstance(inp, Tag): + continue + name = inp.get("name") + if isinstance(name, str): + value = inp.get("value", "") + params[name] = value if isinstance(value, str) else "" + return action, params + + +def _resolve_gdrive_response(session: requests.Session, file_id: str) -> requests.Response: + """Resolve a Google Drive file ID to a streaming response for the actual file. + + Small files download directly; large files first return an HTML + "can't scan for viruses" warning page whose form must be submitted to + obtain the real file. The returned response is opened with ``stream=True``, + so the body is not fetched until the caller iterates it. Callers that only + need the headers (e.g. accessibility checks) must ``close()`` the response. + """ + response = session.get( + PINEAPPLE_GDRIVE_URL, + params={"id": file_id}, + stream=True, + timeout=DEFAULT_REQUESTS_TIMEOUT, + ) + response.raise_for_status() + + content_type = response.headers.get("Content-Type", "") + if "text/html" in content_type: + action, params = _parse_gdrive_form(response.text) + if action: + response = session.get(action, params=params, stream=True, timeout=DEFAULT_REQUESTS_TIMEOUT) + response.raise_for_status() + else: + # Fall back to the legacy cookie-based confirm token + token = next((v for k, v in response.cookies.items() if k.startswith("download_warning")), None) + if token: + response = session.get( + PINEAPPLE_GDRIVE_URL, + params={"id": file_id, "confirm": token}, + stream=True, + timeout=DEFAULT_REQUESTS_TIMEOUT, + ) + response.raise_for_status() + + return response + + +def _download_from_gdrive(file_id: str, dest_path: str, verbose: bool = True) -> None: + """Download a (potentially large) file from Google Drive by file ID.""" + session = requests.Session() + session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; gget)"}) + + response = _resolve_gdrive_response(session, file_id) + + with open(dest_path, "wb") as fh: + for chunk in response.iter_content(chunk_size=32768): + if chunk: + fh.write(chunk) + + if verbose: + logger.info(f"Saved {dest_path}") + + +@overload +def pineapple( + name: str | None = None, + category: _CategoryT = "segmentation", + download: bool = False, + out_dir: str = ".", + save: bool = False, + verbose: bool = True, + *, + json: Literal[True], +) -> list[dict[str, Any]] | None: ... + + +@overload +def pineapple( + name: str | None = None, + category: _CategoryT = "segmentation", + download: bool = False, + out_dir: str = ".", + save: bool = False, + verbose: bool = True, + json: Literal[False] = False, +) -> pd.DataFrame | None: ... + + +def pineapple( + name: str | None = None, + category: str = "segmentation", + download: bool = False, + out_dir: str = ".", + save: bool = False, + verbose: bool = True, + json: bool = False, +) -> pd.DataFrame | list[dict[str, Any]] | None: + """List and download curated bio-imaging datasets and model weights from Pineapple. + + Pineapple (https://github.com/tomouellette/pineapple) curates and standardizes + a collection of bio-imaging datasets (segmentation and benchmark) and + pre-trained self-supervised model weights, hosted on Google Drive. `gget + pineapple` lets you browse this catalog and download the resources directly, + without installing the Pineapple Rust binary. + + Args: + - name Name of the dataset/weights to fetch, e.g. "vicar_2021" or + "dino_vit_small". If None (default), the full catalog for the + chosen 'category' is returned. + - category Resource category: "segmentation", "benchmark", or "weights". + Default: "segmentation". + - download If True (and 'name' is given), download the resource into 'out_dir'. + Default: False. + - out_dir Directory to download the resource into. Default: "." (current directory). + - save If True, save the returned catalog table as csv/json. Default: False. + - verbose True/False whether to print progress information. Default: True. + - json If True, returns results in json format instead of data frame. Default: False. + + Returns a data frame (or list of dicts if json=True) describing the catalog + entry/entries (name, category, authors, size in GB, license, filename, and + Google Drive ID). Please check each dataset's original reference and license + before use. + """ + category = str(category).lower() + if category not in _CATEGORIES: + raise ValueError(f"Invalid category '{category}'. Expected one of: {', '.join(_CATEGORIES)}") + + registry = _CATEGORIES[category] + + if name is None: + rows = [_catalog_row(category, n, info) for n, info in registry.items()] + results_df = pd.DataFrame(rows, columns=_COLUMNS) + if download: + logger.warning("'download' requires a specific 'name'; returning the catalog instead.") + else: + if name not in registry: + raise ValueError(f"'{name}' not found in category '{category}'. Available: {', '.join(registry)}") + info = registry[name] + + if download: + os.makedirs(out_dir, exist_ok=True) + filename = _resource_filename(category, name, info) + dest = os.path.join(out_dir, filename) + if verbose: + logger.info(f"Downloading pineapple {category} resource '{name}' ({info['size_gb']} GB) to {dest}...") + _download_from_gdrive(info["file_id"], dest, verbose) + + results_df = pd.DataFrame([_catalog_row(category, name, info)], columns=_COLUMNS) + + if json: + results_dict = json_package.loads(results_df.to_json(orient="records")) + if save: + with open("gget_pineapple_results.json", "w", encoding="utf-8") as f: + json_package.dump(results_dict, f, ensure_ascii=False, indent=4) + return results_dict + + if save: + results_df.to_csv("gget_pineapple_results.csv", index=False) + + return results_df diff --git a/gget/main.py b/gget/main.py index 7a2944b09..d954d75fd 100644 --- a/gget/main.py +++ b/gget/main.py @@ -37,6 +37,7 @@ from .gget_mutate import mutate # noqa: E402 from .gget_opentargets import OPENTARGETS_RESOURCES, opentargets # noqa: E402 from .gget_pdb import pdb # noqa: E402 +from .gget_pineapple import pineapple # noqa: E402 from .gget_ref import ref # noqa: E402 from .gget_search import search # noqa: E402 from .gget_seq import seq # noqa: E402 @@ -1002,6 +1003,75 @@ def main() -> None: help="DEPRECATED - json is now the default output format (convert to csv using flag [--csv]).", ) + ## gget pineapple subparser + pineapple_desc = "List and download curated bio-imaging datasets and model weights from Pineapple." + parser_pineapple = parent_subparsers.add_parser( + "pineapple", + parents=[parent], + description=pineapple_desc, + help=pineapple_desc, + add_help=True, + formatter_class=CustomHelpFormatter, + ) + parser_pineapple.add_argument( + "name", + type=str, + nargs="?", + default=None, + help="Name of the dataset/weights to fetch, e.g. 'vicar_2021' or 'dino_vit_small'. Omit to list the catalog.", + ) + parser_pineapple.add_argument( + "-c", + "--category", + type=str, + default="segmentation", + choices=["segmentation", "benchmark", "weights"], + required=False, + help="Resource category: 'segmentation', 'benchmark', or 'weights'. Default: 'segmentation'.", + ) + parser_pineapple.add_argument( + "-d", + "--download", + default=False, + action="store_true", + required=False, + help="Download the resource (requires a specific 'name') into --out_dir.", + ) + parser_pineapple.add_argument( + "-od", + "--out_dir", + type=str, + default=".", + required=False, + help="Directory to download the resource into. Default: current directory.", + ) + parser_pineapple.add_argument( + "-csv", + "--csv", + default=True, + action="store_false", + required=False, + help="Returns results in csv format instead of json.", + ) + parser_pineapple.add_argument( + "-o", + "--out", + type=str, + required=False, + help=( + "Path to the file the catalog table will be saved in, e.g. path/to/directory/results.csv (or .json).\n" + "Default: Standard out." + ), + ) + parser_pineapple.add_argument( + "-q", + "--quiet", + default=True, + action="store_false", + required=False, + help="Does not print progress information.", + ) + ## gget enrichr subparser enrichr_desc = "Perform an enrichment analysis on a list of genes using Enrichr." parser_enrichr = parent_subparsers.add_parser( @@ -2956,6 +3026,7 @@ def main() -> None: "muscle": parser_muscle, "blast": parser_blast, "blat": parser_blat, + "pineapple": parser_pineapple, "enrichr": parser_enrichr, "archs4": parser_archs4, "setup": parser_setup, @@ -3502,6 +3573,39 @@ def main() -> None: if not args.out and args.csv: print(json.dumps(gget_results, ensure_ascii=False, indent=4)) + ## pineapple return + if args.command == "pineapple": + pineapple_results = pineapple( + name=args.name, + category=args.category, + download=args.download, + out_dir=args.out_dir, + json=args.csv, + verbose=args.quiet, + ) + + # Check if the function returned something + if pineapple_results is not None: + # Save results if args.out specified + if args.out and not args.csv: + directory = "/".join(args.out.split("/")[:-1]) + if directory != "": + os.makedirs(directory, exist_ok=True) + pineapple_results.to_csv(args.out, index=False) + + if args.out and args.csv: + directory = "/".join(args.out.split("/")[:-1]) + if directory != "": + os.makedirs(directory, exist_ok=True) + with open(args.out, "w", encoding="utf-8") as f: + json.dump(pineapple_results, f, ensure_ascii=False, indent=4) + + # Print results if no directory specified + if not args.out and not args.csv: + pineapple_results.to_csv(sys.stdout, index=False) + if not args.out and args.csv: + print(json.dumps(pineapple_results, ensure_ascii=False, indent=4)) + ## enrichr return if args.command == "enrichr": # Handle deprecated flags for backwards compatibility diff --git a/tests/fixtures/test_pineapple.json b/tests/fixtures/test_pineapple.json new file mode 100644 index 000000000..3d8ed196a --- /dev/null +++ b/tests/fixtures/test_pineapple.json @@ -0,0 +1,18 @@ +{ + "test_pineapple_bad_category": { + "type": "error", + "args": { + "category": "banana" + }, + "expected_result": "ValueError", + "expected_msg": "Invalid category 'banana'. Expected one of: segmentation, benchmark, weights" + }, + "test_pineapple_bad_name": { + "type": "error", + "args": { + "name": "not_a_dataset", + "category": "segmentation" + }, + "expected_result": "ValueError" + } +} diff --git a/tests/test_pineapple.py b/tests/test_pineapple.py new file mode 100644 index 000000000..e0f47a9de --- /dev/null +++ b/tests/test_pineapple.py @@ -0,0 +1,185 @@ +import json +import unittest +from unittest.mock import patch + +import gget.gget_pineapple as gget_pineapple +import requests +from gget.gget_pineapple import ( + _catalog_row, + _parse_gdrive_form, + _resource_filename, + pineapple, +) + +from .from_json import from_json + +with open("./tests/fixtures/test_pineapple.json") as json_file: + pineapple_dict = json.load(json_file) + + +class TestPineapple(unittest.TestCase, metaclass=from_json(pineapple_dict, pineapple)): + pass # tests loaded from json + + +class TestPineappleHelpers(unittest.TestCase): + """Network-free tests of the Pineapple catalog/helpers (issue #161).""" + + def test_list_segmentation(self): + df = pineapple(verbose=False) + self.assertEqual(list(df.columns), gget_pineapple._COLUMNS) + self.assertEqual(df.shape[0], len(gget_pineapple._SEGMENTATION)) + self.assertIn("vicar_2021", set(df["name"])) + # filename is DERIVED (underscore -> dash, + .tar.gz), so this exercises + # real logic -- unlike asserting raw catalog values back at themselves. + row = df[df["name"] == "vicar_2021"].iloc[0] + self.assertEqual(row["filename"], "vicar-2021.tar.gz") + + def test_list_weights(self): + df = pineapple(category="weights", verbose=False) + self.assertEqual(df.shape[0], len(gget_pineapple._WEIGHTS)) + row = df[df["name"] == "dino_vit_small"].iloc[0] + self.assertEqual(row["filename"], "dinov2_vits14_imagenet.safetensors") + + def test_single_entry_json(self): + result = pineapple("vicar_2021", json=True, verbose=False) + self.assertIsInstance(result, list) + self.assertEqual(len(result), 1) + self.assertEqual(result[0]["name"], "vicar_2021") + + def test_resource_filename(self): + self.assertEqual( + _resource_filename("segmentation", "vicar_2021", gget_pineapple._SEGMENTATION["vicar_2021"]), + "vicar-2021.tar.gz", + ) + self.assertEqual( + _resource_filename("weights", "dino_vit_small", gget_pineapple._WEIGHTS["dino_vit_small"]), + "dinov2_vits14_imagenet.safetensors", + ) + + def test_catalog_row_types(self): + row = _catalog_row("benchmark", "runtime", gget_pineapple._BENCHMARK["runtime"]) + self.assertEqual(row["category"], "benchmark") + self.assertIsInstance(row["size_gb"], float) + + def test_parse_gdrive_form(self): + html = ( + '
' + '' + '' + '
' + ) + action, params = _parse_gdrive_form(html) + self.assertEqual(action, "https://drive.usercontent.google.com/download") + self.assertEqual(params, {"id": "ABC123", "confirm": "t", "uuid": "xyz"}) + + def test_parse_gdrive_form_absent(self): + action, params = _parse_gdrive_form("direct download") + self.assertIsNone(action) + self.assertEqual(params, {}) + + @patch.object(gget_pineapple, "_download_from_gdrive") + def test_download_invokes_gdrive(self, mock_dl): + df = pineapple("vicar_2021", download=True, out_dir="/tmp/pineapple_test_dir", verbose=False) + self.assertTrue(mock_dl.called) + args, _ = mock_dl.call_args + # Called with the correct Google Drive file ID and destination path + self.assertEqual(args[0], "12tJOlIHZPFqp8GLek_jV__Uhhgsa530_") + self.assertTrue(args[1].endswith("vicar-2021.tar.gz")) + self.assertEqual(df.iloc[0]["name"], "vicar_2021") + + +class TestPineappleLiveAccess(unittest.TestCase): + """Live data test: verify a representative set of Pineapple resources is + still downloadable from Google Drive (issue #161). + + This hits Google Drive over the network but never downloads the (multi-GB) + bodies. Each file ID is resolved through the *same* production code path as + a real download (``_resolve_gdrive_response``, including the large-file + virus-scan-warning confirmation), and only the response headers are checked. + + Purpose: if someone edits the catalog, or upstream repoints/removes a file, + this fails loudly -- the Google-Drive-reported filename must still match the + catalog and the ID must still resolve to a binary download rather than an + error/quota HTML page. + + Coverage is a small representative sample (not all 30 resources) to keep CI + fast and resilient to Google Drive rate-limiting: one resource per category, + both filename conventions (.tar.gz datasets vs explicit .safetensors + weights), and both resolution paths (direct download vs the large-file + virus-scan-warning form). + """ + + _CATALOGS = { + "segmentation": gget_pineapple._SEGMENTATION, + "benchmark": gget_pineapple._BENCHMARK, + "weights": gget_pineapple._WEIGHTS, + } + + _SAMPLE = [ + ("segmentation", "arvidsson_2022"), # small .tar.gz, direct download + ("benchmark", "kromp_2023"), # small .tar.gz, benchmark category + ("weights", "dino_vit_small"), # explicit .safetensors filename + ("segmentation", "hpa_2022"), # large file -> virus-scan-warning form + ] + + # 1 MB floor: catches an ID repointed to a tiny placeholder/error file. + # NOT tied to the catalog's size_gb, which is only approximate (e.g. + # livecell_2021 lists 3.26 GB but the real file is ~1.81 GB). + _MIN_BYTES = 1_000_000 + + def _check_resource(self, category, name): + info = self._CATALOGS[category][name] + expected = _catalog_row(category, name, info) + + session = requests.Session() + session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; gget)"}) + try: + response = gget_pineapple._resolve_gdrive_response(session, expected["google_drive_id"]) + except requests.RequestException as exc: + self.skipTest(f"Network error reaching Google Drive for {name}: {exc}") + + try: + content_type = response.headers.get("Content-Type", "") + # Google Drive serves a transient HTML "download quota exceeded" page + # under load. Treat that as a skip (not a failure); a genuinely missing + # file would have raised a 404/410 in _resolve_gdrive_response above. + if "text/html" in content_type: + self.skipTest( + f"Google Drive returned an HTML page (likely download-quota " + f"throttling) for {name}; skipping live check." + ) + + self.assertEqual(response.status_code, 200, f"{name}: unexpected status code") + self.assertIn( + "octet-stream", + content_type, + f"{name}: expected a binary download, got Content-Type {content_type!r}", + ) + + disposition = response.headers.get("Content-Disposition", "") + self.assertIn( + f'filename="{expected["filename"]}"', + disposition, + f"{name}: Google Drive filename does not match the catalog " + f"(expected {expected['filename']!r}, Content-Disposition={disposition!r}). " + f"The file ID may have been repointed upstream.", + ) + + length = response.headers.get("Content-Length") + self.assertIsNotNone(length, f"{name}: response is missing Content-Length") + self.assertGreater( + int(length), + self._MIN_BYTES, + f"{name}: file is implausibly small ({length} bytes) -- possible placeholder", + ) + finally: + response.close() + + def test_live_resources_accessible(self): + for category, name in self._SAMPLE: + with self.subTest(category=category, name=name): + self._check_resource(category, name) + + +if __name__ == "__main__": + unittest.main()