diff --git a/docs/conf.py b/docs/conf.py index 47dd1bf02e..c50d6e4298 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -14,6 +14,7 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ + 'sphinxcontrib.mermaid', 'myst_parser', 'sphinx_copybutton', 'sphinxext.rediraffe', diff --git a/docs/dataset-lifecycle.md b/docs/dataset-lifecycle.md new file mode 100644 index 0000000000..c160647a9c --- /dev/null +++ b/docs/dataset-lifecycle.md @@ -0,0 +1,268 @@ +--- +name: Dataset Lifecycle +route: /dataset-lifecycle +--- + +# Dataset Lifecycle + +Datasets are git/git-annex repositories managed by datalad-service. This +document describes the states a dataset, its snapshots, and their DOIs +pass through, and the mutations that drive transitions between them. + +## Dataset states + +A dataset's state is determined by two fields: whether it has any +snapshots and whether it is public. + +```{mermaid} +stateDiagram-v2 + [*] --> Draft + Draft --> Embargoed : createSnapshot() + Embargoed --> Public : publishDataset() + Public --> Embargoed : updatePublic(false) [admin] + Draft --> Deleted : deleteDataset() + Embargoed --> Deleted : deleteDataset() + Public --> Deleted : deleteDataset() +``` + +| State | Determination | DB representation | +| ------------- | -------------------- | -------------------------------------------------- | +| **Draft** | Unversioned, private | `Dataset.public = false`, no `Snapshot` docs exist | +| **Embargoed** | Versioned, private | `Dataset.public = false`, ≥1 `Snapshot` doc exists | +| **Public** | Versioned, public | `Dataset.public = true`, `Dataset.publishDate` set | +| **Deleted** | Dataset removed | `Deletion` doc created | + +These states are implicit — there is no `Dataset.status` field. The code +does not use the term "embargoed"; both Draft and Embargoed datasets have +`public = false` and are distinguished only by the presence of snapshots. + +**Key constraints:** + +- Draft → Embargoed is irreversible. Once a snapshot exists, the dataset + cannot return to Draft. +- Public → Embargoed requires admin privileges. + +### Creation + +`createDataset()` creates a new `Dataset` document with `public: false` +and fires a `{ type: "created" }` event. + +- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` + +### Snapshotting (Draft → Embargoed, or new version) + +`createSnapshot()` tags the current HEAD as a new version. The first +snapshot transitions the dataset from Draft to Embargoed. Subsequent +snapshots do not change the dataset state. + +- **JS:** `packages/openneuro-server/src/datalad/snapshots.ts` +- **Python:** `services/datalad/datalad_service/tasks/snapshots.py` +- **Event:** `{ type: "versioned", version: tag }` + +**Sequence:** + +1. Acquire distributed lock (`lockSnapshot()`, 30-min TTL) +2. Create `"versioned"` event +3. Mint DOI via DataCite MDS API (`createIfNotExistsDoi()`) + — format: `doi:{prefix}/openneuro.{datasetId}.v{tag}` +4. Update `dataset_description.json` with `DatasetDOI` field +5. Update `CHANGES` file with version and changelog entries +6. POST to datalad-service to create git tag +7. Store `Snapshot` document in MongoDB +8. Clear Redis caches +9. Queue for Elasticsearch indexing +10. Notify dataset followers + +**Preconditions:** BIDS validation and git-annex fsck should pass before +creating a snapshot. This is currently enforced in the UI, not in the +`createSnapshot()` API. + +### Publishing (Embargoed → Public) + +`publishDataset()` sets `Dataset.public = true` and triggers export to +S3 and GitHub. + +- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/publish.ts` +- **JS function:** `packages/openneuro-server/src/datalad/dataset.ts` — `updatePublic()` +- **Python task:** `services/datalad/datalad_service/tasks/publish.py` +- **Event:** `{ type: "published", public: true }` + +**Sequence:** + +1. Check write permissions +2. Set `Dataset.public = true` and `Dataset.publishDate = new Date()` +3. Create `"published"` event +4. POST to datalad-service `/datasets/{id}/publish` +5. Create S3 and GitHub remotes (`create_remotes_and_export()`) +6. Export dataset to S3-PUBLIC and GitHub (`export_dataset()`) +7. Set S3 access tags to "public" +8. Run remote fsck to verify exported data +9. Drop local annexed files after verification + +### Re-embargo (Public → Embargoed) + +`updatePublic(datasetId, false, user)` sets `Dataset.public = false` and +updates S3 access tags to "private". + +- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` — `updatePublic()` +- **Event:** `{ type: "published", public: false }` + +**Known issue:** This also sets `Dataset.publishDate = new Date()` rather +than clearing it. + +### Deletion + +`deleteDataset()` removes the dataset from the git backend and +Elasticsearch. Currently, datasets with snapshots require admin action to +delete. + +- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` — `deleteDataset()` +- **Event:** `{ type: "deleted" }` +- Creates a `Deletion` document (datasetId, user, reason, optional + redirect URL) + +## Snapshot status + +Snapshots can be independently deprecated without changing the dataset's +state. + +```{mermaid} +stateDiagram-v2 + [*] --> Active : createSnapshot() + Active --> Deprecated : deprecateSnapshot() + Deprecated --> Active : undoDeprecateSnapshot() +``` + +| Status | DB representation | +| -------------- | ----------------------------------------------------------- | +| **Active** | `Snapshot` doc exists, no matching `DeprecatedSnapshot` doc | +| **Deprecated** | `DeprecatedSnapshot` doc with user, reason, timestamp | + +- **JS:** `packages/openneuro-server/src/graphql/resolvers/snapshots.ts` + +## DOI assignment + +Each snapshot is assigned a DOI via the DataCite MDS API at creation +time. DOIs are minted immediately as findable — there is no state +management after creation. + +| Field | Value | +| ------- | ------------------------------------------- | +| Format | `doi:{prefix}/openneuro.{datasetId}.v{tag}` | +| API | DataCite MDS (XML metadata + `PUT /doi`) | +| Created | During `createSnapshot()` (step 3) | +| State | Not tracked locally | + +- **JS:** `packages/openneuro-server/src/libs/doi/index.ts` +- **Handler:** `packages/openneuro-server/src/handlers/doi.ts` +- **Model:** `packages/openneuro-server/src/models/doi.ts` + — fields: `datasetId`, `snapshotId`, `doi` + +DOI metadata includes creators, title, publisher ("Openneuro"), +publication year, and resource type. The DOI string is written into +`dataset_description.json` before the git tag is created. + +DOI state is never updated after creation. The DOI remains findable at +DataCite regardless of whether the dataset is later re-embargoed, the +snapshot is deprecated, or the dataset is deleted. + +## Working tree + +The working tree (git HEAD) can diverge from the latest snapshot when +users upload or edit files. Draft datasets are always diverged (no +snapshot exists). For Embargoed and Public datasets: + +| Status | Determination | +| ------------ | ------------------------------- | +| **Clean** | `HEAD == latestSnapshot.hexsha` | +| **Diverged** | `HEAD != latestSnapshot.hexsha` | + +Divergence does not change the dataset's lifecycle state. A new +`createSnapshot()` call is required to tag the current HEAD. + +- **JS:** `packages/openneuro-server/src/datalad/draft.ts` + — `getDraftRevision()`, `getDraftInfo()`, `resetDraft()` + +### Data retention + +When the working tree diverges from the latest snapshot, a notification +schedule enforces data retention: + +- 14 days diverged: first warning email +- 21 days: second warning +- 28 days: deletion notice +- No snapshot within 24h of first upload: one-time reminder + +- **JS:** `packages/openneuro-server/src/datalad/dataRetentionNotifications.ts` + +## Validation + +Validation is a precondition for snapshotting, not a persistent state. +Two checks are involved: + +### BIDS schema validation + +Runs the BIDS validator (deno-based, v2.3.2) against a specific commit. + +- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/validation.ts` — `revalidate()` +- **Python task:** `services/datalad/datalad_service/tasks/validator.py` — `validate_dataset()` +- **Storage:** `Validation` collection + +### git-annex fsck + +Checks data consistency of annexed files. + +- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/fileCheck.ts` — `fsckDataset()` +- **Python task:** `services/datalad/datalad_service/tasks/fsck.py` — `git_annex_fsck_local()` +- Runs `git-annex fsck -J4` with incremental 45-day schedule +- **Storage:** `FileCheck` collection + +## Data model reference + +### MongoDB collections + +| Collection | File | Key fields | +| -------------------- | ------------------------------------------- | ---------------------------------------------- | +| `Dataset` | `packages/.../models/dataset.ts` | `id, public, publishDate, created, modified` | +| `Snapshot` | `packages/.../models/snapshot.ts` | `datasetId, tag, hexsha, created` | +| `DeprecatedSnapshot` | `packages/.../models/deprecatedSnapshot.ts` | `id, user, reason, timestamp` | +| `DatasetEvent` | `packages/.../models/datasetEvents.ts` | `datasetId, userId, event, success, timestamp` | +| `Doi` | `packages/.../models/doi.ts` | `datasetId, snapshotId, doi` | +| `Validation` | `packages/.../models/validation.ts` | `id (ref), datasetId, issues, codeMessages` | +| `FileCheck` | `packages/.../models/fileCheck.ts` | `datasetId, hexsha, refs, annexFsck[]` | +| `Deletion` | `packages/.../models/deletion.ts` | `datasetId, user, reason, redirect` | + +All model files are under `packages/openneuro-server/src/models/`. + +### GraphQL types + +Defined in `packages/openneuro-server/src/graphql/schema.ts`: + +- `Dataset` — core entity with `public`, `publishDate`, `draft`, `snapshots` +- `Draft` — working tree with `modified`, `summary`, `validation`, `fileCheck` +- `Snapshot` — tagged version with `tag`, `hexsha`, `deprecated` +- `DeprecatedSnapshot` — deprecation record +- `DatasetValidation` — validation results with error/warning counts + +### Mutations + +| Mutation | Effect | +| ----------------------- | --------------------------------------------------- | +| `createDataset` | Creates dataset in Draft state | +| `createSnapshot` | Creates snapshot + DOI; Draft → Embargoed on first | +| `publishDataset` | Embargoed → Public; exports to S3/GitHub | +| `updatePublic(false)` | Public → Embargoed (admin only) | +| `deprecateSnapshot` | Marks a snapshot as deprecated | +| `undoDeprecateSnapshot` | Removes deprecation | +| `deleteDataset` | Removes dataset (admin required if snapshots exist) | +| `fsckDataset` | Triggers git-annex fsck | +| `revalidate` | Triggers BIDS validation | + +### Event types + +Dataset events are stored in the `DatasetEvent` collection and track all +state transitions: + +`created` · `versioned` · `deleted` · `published` · `permissionChange` · +`git` · `upload` · `note` · `contributorRequest` · `contributorCitation` · +`contributorRequestResponse` · `contributorCitationResponse` diff --git a/docs/index.md b/docs/index.md index 16d778b69e..f88054ae8b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -90,6 +90,7 @@ structure of our package and hosting setup may be interested in the following se :maxdepth: 1 architecture +dataset-lifecycle maintenance Source repository (GitHub) ``` diff --git a/docs/requirements.txt b/docs/requirements.txt index 2146ad7fd7..0b008834f7 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -4,3 +4,4 @@ myst-parser >=0.18.1 sphinx-copybutton >=0.5.0 pygments-graphql-lexer >=0.1.0 sphinxext-rediraffe >=0.2.7 +sphinxcontrib-mermaid >=2.0.1