Skip to content

feat: add dataset collections - #177

Merged
hweej merged 6 commits into
mainfrom
feat/collections
Aug 13, 2026
Merged

feat: add dataset collections#177
hweej merged 6 commits into
mainfrom
feat/collections

Conversation

@hweej

@hweej hweej commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Groups datasets under the study they came from. Motivated by the 71 HTAN datasets recently added to the catalog, which made a flat dataset list unusable — 32 of them are samples from a single breast-cancer study, and there was nowhere to record what a study is or link to its paper.

Backend only. The frontend work is a separate change.

Design

  • A collection is a first-class entity with a name, slug, description, and publication link.
  • A dataset belongs to at most one collection, and collections do not nest. datasets.collection_id is nullable, so datasets predating collections stay valid with no data migration.
  • Collections carry no access fields. Visibility is derived: a collection is returned only when the caller can access at least one dataset in it, and dataset_count counts only accessible datasets. user_can_access stays the single access rule, so the two can never disagree.
  • A collection with no accessible datasets returns 404, not 403 — a 403 would confirm a gated study exists. The unknown-slug and exists-but-inaccessible responses are byte-identical, and the detail query is scoped by collection_id so the two paths don't differ measurably in time either.
  • ondelete="SET NULL" on datasets.collection_id: deleting a collection orphans its datasets, never deletes them.

What's here

Commit
7a49b55 Collection model, datasets.collection_id, Alembic revision
68e27d5 da1f4d6 collection on dataset responses, eagerly loaded via outerjoin
8d917f6 e7de9d4 GET /api/collections and /api/collections/{slug}
5e50e80 Admin CRUD, collection_id on dataset create/update

GET /api/datasets is unchanged apart from the additive collection field, and collection_id is null everywhere until a follow-up backfill assigns the HTAN datasets.

Testing

make test342 passed (321 before this branch; 21 new). Covers derived visibility for anonymous and role-holding callers, partial access, the null-collection path, migration up and down, and that datasets survive collection deletion.

Note: three admin-key tests fail if ADMIN_API_KEY is exported in your shell — it leaks into Settings. Run env -u ADMIN_API_KEY make test. That's pre-existing and unrelated to this branch.

Follow-ups, not in this PR

  • Backfill script to create the 8 HTAN collections and assign the 71 datasets.
  • Frontend: collections tab, /collections/:slug page.
  • The migration autogenerate surfaced pre-existing model/schema drift on chat_threads/chat_messages indexes and Dataset.prompt_addendum's type. Deliberately excluded here, but it's real and worth a separate look.

hweej added 6 commits August 13, 2026 06:59
- Add tests for the authenticated half of derived visibility: a caller
  with the matching role sees a previously-gated collection and the
  larger dataset_count/dataset list on a mixed collection.
- Scope _accessible_by_collection's dataset query to the resolved
  collection id on the detail path, so an existing-but-inaccessible
  collection no longer runs a full dataset-table scan that an unknown
  slug skips -- closing a timing side channel that could otherwise
  reveal collection existence.
@hweej
hweej merged commit b57c781 into main Aug 13, 2026
3 checks passed
@hweej
hweej deleted the feat/collections branch August 13, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant