From 30919bc959a3ff1794190eafad77583b4686ca4c Mon Sep 17 00:00:00 2001 From: "Christopher J. Markiewicz" Date: Mon, 6 Apr 2026 11:40:42 -0400 Subject: [PATCH 1/2] doc: Document current dataset lifecycle, propose DOI lifecycle --- docs/dataset-lifecycle.md | 268 ++++++++++++++++++ docs/specs/2026-04-06-doi-lifecycle-design.md | 221 +++++++++++++++ 2 files changed, 489 insertions(+) create mode 100644 docs/dataset-lifecycle.md create mode 100644 docs/specs/2026-04-06-doi-lifecycle-design.md diff --git a/docs/dataset-lifecycle.md b/docs/dataset-lifecycle.md new file mode 100644 index 0000000000..f19295f786 --- /dev/null +++ b/docs/dataset-lifecycle.md @@ -0,0 +1,268 @@ +--- +name: Dataset Lifecycle +route: /dataset-lifecycle +--- + +# Dataset Lifecycle + +Datasets are git/git-annex repositories managed by datalad-service. This +document describes the states a dataset, its snapshots, and their DOIs +pass through, and the mutations that drive transitions between them. + +## Dataset states + +A dataset's state is determined by two fields: whether it has any +snapshots and whether it is public. + +```mermaid +stateDiagram-v2 + [*] --> Draft + Draft --> Embargoed : createSnapshot() + Embargoed --> Public : publishDataset() + Public --> Embargoed : updatePublic(false) [admin] + Draft --> Deleted : deleteDataset() + Embargoed --> Deleted : deleteDataset() + Public --> Deleted : deleteDataset() +``` + +| State | Determination | DB representation | +| ------------- | -------------------- | -------------------------------------------------- | +| **Draft** | Unversioned, private | `Dataset.public = false`, no `Snapshot` docs exist | +| **Embargoed** | Versioned, private | `Dataset.public = false`, ≥1 `Snapshot` doc exists | +| **Public** | Versioned, public | `Dataset.public = true`, `Dataset.publishDate` set | +| **Deleted** | Dataset removed | `Deletion` doc created | + +These states are implicit — there is no `Dataset.status` field. The code +does not use the term "embargoed"; both Draft and Embargoed datasets have +`public = false` and are distinguished only by the presence of snapshots. + +**Key constraints:** + +- Draft → Embargoed is irreversible. Once a snapshot exists, the dataset + cannot return to Draft. +- Public → Embargoed requires admin privileges. + +### Creation + +`createDataset()` creates a new `Dataset` document with `public: false` +and fires a `{ type: "created" }` event. + +- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` + +### Snapshotting (Draft → Embargoed, or new version) + +`createSnapshot()` tags the current HEAD as a new version. The first +snapshot transitions the dataset from Draft to Embargoed. Subsequent +snapshots do not change the dataset state. + +- **JS:** `packages/openneuro-server/src/datalad/snapshots.ts` +- **Python:** `services/datalad/datalad_service/tasks/snapshots.py` +- **Event:** `{ type: "versioned", version: tag }` + +**Sequence:** + +1. Acquire distributed lock (`lockSnapshot()`, 30-min TTL) +2. Create `"versioned"` event +3. Mint DOI via DataCite MDS API (`createIfNotExistsDoi()`) + — format: `doi:{prefix}/openneuro.{datasetId}.v{tag}` +4. Update `dataset_description.json` with `DatasetDOI` field +5. Update `CHANGES` file with version and changelog entries +6. POST to datalad-service to create git tag +7. Store `Snapshot` document in MongoDB +8. Clear Redis caches +9. Queue for Elasticsearch indexing +10. Notify dataset followers + +**Preconditions:** BIDS validation and git-annex fsck should pass before +creating a snapshot. This is currently enforced in the UI, not in the +`createSnapshot()` API. + +### Publishing (Embargoed → Public) + +`publishDataset()` sets `Dataset.public = true` and triggers export to +S3 and GitHub. + +- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/publish.ts` +- **JS function:** `packages/openneuro-server/src/datalad/dataset.ts` — `updatePublic()` +- **Python task:** `services/datalad/datalad_service/tasks/publish.py` +- **Event:** `{ type: "published", public: true }` + +**Sequence:** + +1. Check write permissions +2. Set `Dataset.public = true` and `Dataset.publishDate = new Date()` +3. Create `"published"` event +4. POST to datalad-service `/datasets/{id}/publish` +5. Create S3 and GitHub remotes (`create_remotes_and_export()`) +6. Export dataset to S3-PUBLIC and GitHub (`export_dataset()`) +7. Set S3 access tags to "public" +8. Run remote fsck to verify exported data +9. Drop local annexed files after verification + +### Re-embargo (Public → Embargoed) + +`updatePublic(datasetId, false, user)` sets `Dataset.public = false` and +updates S3 access tags to "private". + +- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` — `updatePublic()` +- **Event:** `{ type: "published", public: false }` + +**Known issue:** This also sets `Dataset.publishDate = new Date()` rather +than clearing it. + +### Deletion + +`deleteDataset()` removes the dataset from the git backend and +Elasticsearch. Currently, datasets with snapshots require admin action to +delete. + +- **JS:** `packages/openneuro-server/src/datalad/dataset.ts` — `deleteDataset()` +- **Event:** `{ type: "deleted" }` +- Creates a `Deletion` document (datasetId, user, reason, optional + redirect URL) + +## Snapshot status + +Snapshots can be independently deprecated without changing the dataset's +state. + +```mermaid +stateDiagram-v2 + [*] --> Active : createSnapshot() + Active --> Deprecated : deprecateSnapshot() + Deprecated --> Active : undoDeprecateSnapshot() +``` + +| Status | DB representation | +| -------------- | ----------------------------------------------------------- | +| **Active** | `Snapshot` doc exists, no matching `DeprecatedSnapshot` doc | +| **Deprecated** | `DeprecatedSnapshot` doc with user, reason, timestamp | + +- **JS:** `packages/openneuro-server/src/graphql/resolvers/snapshots.ts` + +## DOI assignment + +Each snapshot is assigned a DOI via the DataCite MDS API at creation +time. DOIs are minted immediately as findable — there is no state +management after creation. + +| Field | Value | +| ------- | ------------------------------------------- | +| Format | `doi:{prefix}/openneuro.{datasetId}.v{tag}` | +| API | DataCite MDS (XML metadata + `PUT /doi`) | +| Created | During `createSnapshot()` (step 3) | +| State | Not tracked locally | + +- **JS:** `packages/openneuro-server/src/libs/doi/index.ts` +- **Handler:** `packages/openneuro-server/src/handlers/doi.ts` +- **Model:** `packages/openneuro-server/src/models/doi.ts` + — fields: `datasetId`, `snapshotId`, `doi` + +DOI metadata includes creators, title, publisher ("Openneuro"), +publication year, and resource type. The DOI string is written into +`dataset_description.json` before the git tag is created. + +DOI state is never updated after creation. The DOI remains findable at +DataCite regardless of whether the dataset is later re-embargoed, the +snapshot is deprecated, or the dataset is deleted. + +## Working tree + +The working tree (git HEAD) can diverge from the latest snapshot when +users upload or edit files. Draft datasets are always diverged (no +snapshot exists). For Embargoed and Public datasets: + +| Status | Determination | +| ------------ | ------------------------------- | +| **Clean** | `HEAD == latestSnapshot.hexsha` | +| **Diverged** | `HEAD != latestSnapshot.hexsha` | + +Divergence does not change the dataset's lifecycle state. A new +`createSnapshot()` call is required to tag the current HEAD. + +- **JS:** `packages/openneuro-server/src/datalad/draft.ts` + — `getDraftRevision()`, `getDraftInfo()`, `resetDraft()` + +### Data retention + +When the working tree diverges from the latest snapshot, a notification +schedule enforces data retention: + +- 14 days diverged: first warning email +- 21 days: second warning +- 28 days: deletion notice +- No snapshot within 24h of first upload: one-time reminder + +- **JS:** `packages/openneuro-server/src/datalad/dataRetentionNotifications.ts` + +## Validation + +Validation is a precondition for snapshotting, not a persistent state. +Two checks are involved: + +### BIDS schema validation + +Runs the BIDS validator (deno-based, v2.3.2) against a specific commit. + +- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/validation.ts` — `revalidate()` +- **Python task:** `services/datalad/datalad_service/tasks/validator.py` — `validate_dataset()` +- **Storage:** `Validation` collection + +### git-annex fsck + +Checks data consistency of annexed files. + +- **JS resolver:** `packages/openneuro-server/src/graphql/resolvers/fileCheck.ts` — `fsckDataset()` +- **Python task:** `services/datalad/datalad_service/tasks/fsck.py` — `git_annex_fsck_local()` +- Runs `git-annex fsck -J4` with incremental 45-day schedule +- **Storage:** `FileCheck` collection + +## Data model reference + +### MongoDB collections + +| Collection | File | Key fields | +| -------------------- | ------------------------------------------- | ---------------------------------------------- | +| `Dataset` | `packages/.../models/dataset.ts` | `id, public, publishDate, created, modified` | +| `Snapshot` | `packages/.../models/snapshot.ts` | `datasetId, tag, hexsha, created` | +| `DeprecatedSnapshot` | `packages/.../models/deprecatedSnapshot.ts` | `id, user, reason, timestamp` | +| `DatasetEvent` | `packages/.../models/datasetEvents.ts` | `datasetId, userId, event, success, timestamp` | +| `Doi` | `packages/.../models/doi.ts` | `datasetId, snapshotId, doi` | +| `Validation` | `packages/.../models/validation.ts` | `id (ref), datasetId, issues, codeMessages` | +| `FileCheck` | `packages/.../models/fileCheck.ts` | `datasetId, hexsha, refs, annexFsck[]` | +| `Deletion` | `packages/.../models/deletion.ts` | `datasetId, user, reason, redirect` | + +All model files are under `packages/openneuro-server/src/models/`. + +### GraphQL types + +Defined in `packages/openneuro-server/src/graphql/schema.ts`: + +- `Dataset` — core entity with `public`, `publishDate`, `draft`, `snapshots` +- `Draft` — working tree with `modified`, `summary`, `validation`, `fileCheck` +- `Snapshot` — tagged version with `tag`, `hexsha`, `deprecated` +- `DeprecatedSnapshot` — deprecation record +- `DatasetValidation` — validation results with error/warning counts + +### Mutations + +| Mutation | Effect | +| ----------------------- | --------------------------------------------------- | +| `createDataset` | Creates dataset in Draft state | +| `createSnapshot` | Creates snapshot + DOI; Draft → Embargoed on first | +| `publishDataset` | Embargoed → Public; exports to S3/GitHub | +| `updatePublic(false)` | Public → Embargoed (admin only) | +| `deprecateSnapshot` | Marks a snapshot as deprecated | +| `undoDeprecateSnapshot` | Removes deprecation | +| `deleteDataset` | Removes dataset (admin required if snapshots exist) | +| `fsckDataset` | Triggers git-annex fsck | +| `revalidate` | Triggers BIDS validation | + +### Event types + +Dataset events are stored in the `DatasetEvent` collection and track all +state transitions: + +`created` · `versioned` · `deleted` · `published` · `permissionChange` · +`git` · `upload` · `note` · `contributorRequest` · `contributorCitation` · +`contributorRequestResponse` · `contributorCitationResponse` diff --git a/docs/specs/2026-04-06-doi-lifecycle-design.md b/docs/specs/2026-04-06-doi-lifecycle-design.md new file mode 100644 index 0000000000..75af34bb28 --- /dev/null +++ b/docs/specs/2026-04-06-doi-lifecycle-design.md @@ -0,0 +1,221 @@ +# DOI Lifecycle State Management — Design Spec + +**Date**: 2026-04-06 +**Status**: Drafted + +## 1. Problem Statement + +OpenNeuro mints DOIs for every snapshot via DataCite's legacy MDS API. The current implementation has three problems: + +1. **DOIs are immediately findable.** A snapshot on an embargoed (private) dataset gets a DOI that is discoverable in DataCite search, leaking the existence of unreleased data. + +2. **DOI state is never updated after creation.** Publishing, deprecating, re-embargoing, or deleting a dataset has no effect on its DOIs at DataCite. A deleted dataset's DOIs remain findable indefinitely. + +3. **Deletion is unnecessarily restrictive.** Any dataset with snapshots requires admin intervention to delete, even when all DOIs are still in draft state and can be cleanly removed from DataCite. + +The `Doi` model (`packages/openneuro-server/src/models/doi.ts`) stores `{ datasetId, snapshotId, doi }` with no state tracking. The DOI creation code (`packages/openneuro-server/src/libs/doi/index.ts`) uses the MDS API, which does not support state transitions. + +## 2. Design Goals + +- **Align DOI discoverability with dataset visibility.** Embargoed datasets get draft DOIs; public datasets get findable DOIs; deprecated snapshots get registered (resolvable but not discoverable) DOIs. +- **Fail fast on metadata issues.** Create DOIs in draft state at snapshot time so invalid metadata is caught before the git tag is created. +- **Reduce admin burden.** Let owners delete their own embargoed datasets when all DOIs are still draft. +- **Migrate existing DOIs.** Backfill state for all existing DOI records and reconcile with DataCite. +- **Handle DataCite failures gracefully.** Detect and recover from state divergence between the local database and DataCite. + +## 3. Architecture + +### System components + +DOI state management spans three layers: + +| Layer | Component | Role | +| ----------------- | ------------------------------------------------------------------- | -------------------------------------- | +| GraphQL resolvers | `packages/openneuro-server/src/graphql/resolvers/publish.ts` et al. | Entry points for lifecycle mutations | +| Dataset service | `packages/openneuro-server/src/datalad/dataset.ts` | Orchestrates dataset state changes | +| DOI service | `packages/openneuro-server/src/libs/doi/index.ts` | Communicates with DataCite API | +| Snapshot service | `packages/openneuro-server/src/datalad/snapshots.ts` | Creates snapshots and mints draft DOIs | +| DOI model | `packages/openneuro-server/src/models/doi.ts` | MongoDB persistence for DOI records | +| Datalad service | `services/datalad/datalad_service/tasks/publish.py` | S3/GitHub export on publish | +| Datalad snapshots | `services/datalad/datalad_service/tasks/snapshots.py` | Git tag creation | + +### Key change: MDS API to REST API + +The current DOI code uses DataCite's MDS API (XML metadata, `PUT /doi`). DOI state transitions (`publish`, `register`, `hide`) require the REST API (`POST/PUT https://api.datacite.org/dois` with JSON payloads). This is the most significant infrastructure change. + +### Data flow + +DOI side effects are injected into existing mutation handlers. No new mutations are added. The DOI service gains three new operations: `publishDoi()`, `hideDoi()`, and `deleteDoi()`, corresponding to DataCite REST API events. + +## 4. DOI State Model + +DataCite defines three DOI states: + +| DOI State | Resolvable | Discoverable | Deletable | When used | +| -------------- | ---------- | ------------ | --------- | --------------------------------------------- | +| **Draft** | No | No | Yes | Embargoed dataset (any snapshot status) | +| **Registered** | Yes | No | No | Public + deprecated snapshot, or re-embargoed | +| **Findable** | Yes | Yes | No | Public dataset, active snapshot | + +### State mapping: dataset lifecycle x snapshot status to DOI state + +| Dataset lifecycle | Snapshot status | DOI state | +| ------------------------ | --------------- | ---------- | +| Embargoed | Active | Draft | +| Embargoed | Deprecated | Draft | +| Public | Active | Findable | +| Public | Deprecated | Registered | +| Embargoed (was Public) | Active | Registered | +| Embargoed (was Public) | Deprecated | Registered | +| Deleted (all DOIs draft) | -- | Deleted | +| Deleted (any non-draft) | -- | Registered | + +### State diagram + +```mermaid +stateDiagram-v2 + [*] --> Draft : createSnapshot() + Draft --> Draft : deprecateSnapshot() / undoDeprecateSnapshot() + Draft --> Findable : publishDataset() + Draft --> [*] : deleteDataset() + Registered --> Findable : publishDataset() / undoDeprecateSnapshot() + Findable --> Registered : updatePublic(false) [admin] / deprecateSnapshot() / deleteDataset() +``` + +### Transition table + +| Trigger | DataCite event | DOI transition | +| ---------------------------------------- | -------------- | ---------------------------- | +| `publishDataset()` | `publish` | Draft/Registered -> Findable | +| `updatePublic(false)` | `hide` | Findable -> Registered | +| `deprecateSnapshot()` (if Public) | `hide` | Findable -> Registered | +| `deprecateSnapshot()` (if Embargoed) | -- | stays Draft | +| `undoDeprecateSnapshot()` (if Public) | `publish` | Registered -> Findable | +| `undoDeprecateSnapshot()` (if Embargoed) | -- | stays Draft | +| `deleteDataset()` (draft DOIs) | DELETE | Draft -> Deleted | +| `deleteDataset()` (findable DOIs) | `hide` | Findable -> Registered | +| `deleteDataset()` (registered DOIs) | -- | stays Registered | + +## 5. Mutation Side Effects + +For each GraphQL mutation that now requires DOI side effects: + +| Mutation | Current behavior | New DOI side effect | Scope | +| ----------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------- | +| `createSnapshot` | Mints DOI via MDS API (findable) | Create DOI in **draft** via REST API. If dataset is already public, immediately transition to **findable**. | Single DOI | +| `publishDataset` | Sets `public=true`, exports to S3 | Call `publish` on all snapshot DOIs. Update local state to **findable**. | All DOIs | +| `updatePublic(false)` | Sets `public=false` | Call `hide` on all snapshot DOIs. Update local state to **registered**. | All DOIs | +| `deprecateSnapshot` | Creates DeprecatedSnapshot doc | If dataset is public: call `hide` on this snapshot's DOI, set state to **registered**. If embargoed: no-op. | Single DOI | +| `undoDeprecateSnapshot` | Removes DeprecatedSnapshot doc | If dataset is public: call `publish` on this snapshot's DOI, set state to **findable**. If embargoed: no-op. | Single DOI | +| `deleteDataset` | Removes dataset | For draft DOIs: DELETE at DataCite. For findable DOIs: `hide`. For registered DOIs: no-op. | All DOIs | +| `createDataset` | Creates empty dataset | No DOI side effect. | -- | + +Side effects execute **after** the primary mutation succeeds but **within the same logical operation**. A DataCite failure on `createSnapshot` is fatal (snapshot creation rolls back). DataCite failures on other mutations are logged and queued for retry (see section 8). + +## 6. Deletion Permissions + +The new rule: dataset owners can delete if all associated DOIs are in draft state. Admin is required when any DOI has left draft. + +| Dataset state | DOI state of snapshots | Who can delete | DOI action at DataCite | +| ------------- | ----------------------- | -------------- | ----------------------------- | +| Draft | No snapshots exist | Owner | N/A | +| Embargoed | All DOIs in draft | Owner | DELETE all DOIs | +| Any | Any registered/findable | Admin only | `hide` findable -> registered | + +**Implementation**: The `deleteDataset()` function in `packages/openneuro-server/src/datalad/dataset.ts` currently checks for admin permissions when snapshots exist. This check changes to: query the `Doi` collection for any records with `state != "draft"` for the dataset. If none found, owner permission suffices. If any found, require admin. + +## 7. Migration + +### Scope + +All existing `Doi` records lack a `state` field. Based on the legacy MDS API behavior, all existing DOIs are expected to be in **findable** state at DataCite. + +### Migration steps + +1. **Add `state` field to Doi model** with no default (allows distinguishing migrated from unmigrated records). +2. **Query DataCite REST API** for each existing DOI to confirm its actual state. +3. **Update local records** with the state reported by DataCite. +4. **Flag discrepancies** -- any DOI not found at DataCite or in an unexpected state is logged for manual review. +5. **For deleted datasets** that still have DOI records: if the DOI is findable at DataCite, transition it to registered via `hide`. + +### Execution + +- Run as a one-time script against the production database. +- Dry-run mode first: report what would change without writing. +- Rate-limit DataCite API calls to avoid hitting quotas. +- Expected volume: one API call per existing DOI record. The DataCite REST API supports bulk lookup but individual GETs are simpler to implement and debug. + +### Rollback + +The migration only adds data (the `state` field). Rolling back means removing the field. No existing data is modified or deleted. + +## 8. Error Handling + +### DataCite unavailable + +| Scenario | Behavior | +| ------------------------------------------ | ------------------------------------------------------------------------ | +| DataCite down during `createSnapshot` | **Fatal.** Snapshot creation fails. No git tag created. User sees error. | +| DataCite down during `publishDataset` | **Non-fatal.** Dataset publishes. DOI transition queued for retry. | +| DataCite down during `deleteDataset` | **Non-fatal.** Dataset deleted locally. DOI cleanup queued for retry. | +| DataCite down during deprecate/undeprecate | **Non-fatal.** Local state updated. DOI transition queued for retry. | + +Rationale: `createSnapshot` is fatal because the DOI string must be embedded in `dataset_description.json` before the git tag. All other mutations have a primary effect (visibility change, deletion) that should not be blocked by a DOI service outage. + +### Metadata validation failure + +DataCite rejects DOI creation if required metadata is missing (e.g., no creators). This fails the `createSnapshot` call with a descriptive error. The user must fix metadata before retrying. + +### Local/remote state divergence + +A reconciliation job should run periodically (e.g., daily cron) to: + +1. Query all local `Doi` records. +2. Compare local `state` with DataCite's reported state. +3. Log any mismatches. +4. Optionally auto-correct by issuing the appropriate DataCite event to match the expected state derived from the dataset lifecycle + snapshot status mapping. + +This handles edge cases like a `hide` call that succeeded at DataCite but failed to update MongoDB, or manual DataCite console changes. + +### Retry strategy + +Failed non-fatal DOI transitions are stored in a retry queue (e.g., a MongoDB collection or Redis list). A background worker retries with exponential backoff, capped at 6 retries over 24 hours. After exhausting retries, the failure is escalated to admin review. + +## 9. Testing Strategy + +### Unit tests + +- **State transition logic**: Given (dataset lifecycle, snapshot status), assert the correct DOI state and DataCite event. Cover all cells in the state mapping table. +- **Deletion permission logic**: Given a set of DOI states for a dataset, assert whether owner or admin permission is required. +- **No DataCite calls for embargoed deprecate/undeprecate**: Verify that no API call is made when the dataset is not public. + +### Integration tests (DataCite test environment) + +DataCite provides a test API at `https://api.test.datacite.org`. Integration tests should: + +- Create a DOI in draft state and verify it is not resolvable. +- Transition draft -> findable via `publish` and verify discoverability. +- Transition findable -> registered via `hide` and verify it resolves but is not discoverable. +- Delete a draft DOI and verify it no longer exists. +- Attempt to delete a registered DOI and verify it fails. +- Submit invalid metadata and verify the error response. + +### Migration dry-run + +- Run the migration script in dry-run mode against a staging database with production data. +- Verify that all DOI records receive a state matching DataCite's reported state. +- Verify that DOIs not found at DataCite are flagged, not silently skipped. + +### End-to-end smoke tests + +- Full lifecycle: create dataset -> snapshot -> publish -> deprecate snapshot -> undeprecate -> re-embargo -> delete. Verify DOI state at each step matches the state mapping table. + +## 10. Open Questions + +The lifecycle document resolved the major design decisions. Remaining implementation-level questions: + +1. **Retry infrastructure**: Use an existing job queue (if one exists in the codebase) or add a new retry mechanism? This is an implementation choice, not a design decision. + +2. **Reconciliation frequency**: Daily reconciliation is suggested, but the appropriate interval depends on operational experience with DataCite reliability. + +3. **`publishDate` bug**: The lifecycle doc notes that `updatePublic(false)` overwrites `publishDate` with the current date rather than clearing it (Gap 4). Should this be fixed as part of this work or tracked separately? From fe6b41eccf37996a9ceb3dd39b15184cc8ba2d1c Mon Sep 17 00:00:00 2001 From: "Christopher J. Markiewicz" Date: Mon, 6 Apr 2026 12:01:20 -0400 Subject: [PATCH 2/2] doc: Render mermaid --- docs/conf.py | 1 + docs/dataset-lifecycle.md | 4 +- docs/index.md | 1 + docs/requirements.txt | 1 + docs/specs/2026-04-06-doi-lifecycle-design.md | 221 ------------------ 5 files changed, 5 insertions(+), 223 deletions(-) delete mode 100644 docs/specs/2026-04-06-doi-lifecycle-design.md diff --git a/docs/conf.py b/docs/conf.py index 47dd1bf02e..c50d6e4298 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -14,6 +14,7 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ + 'sphinxcontrib.mermaid', 'myst_parser', 'sphinx_copybutton', 'sphinxext.rediraffe', diff --git a/docs/dataset-lifecycle.md b/docs/dataset-lifecycle.md index f19295f786..c160647a9c 100644 --- a/docs/dataset-lifecycle.md +++ b/docs/dataset-lifecycle.md @@ -14,7 +14,7 @@ pass through, and the mutations that drive transitions between them. A dataset's state is determined by two fields: whether it has any snapshots and whether it is public. -```mermaid +```{mermaid} stateDiagram-v2 [*] --> Draft Draft --> Embargoed : createSnapshot() @@ -126,7 +126,7 @@ delete. Snapshots can be independently deprecated without changing the dataset's state. -```mermaid +```{mermaid} stateDiagram-v2 [*] --> Active : createSnapshot() Active --> Deprecated : deprecateSnapshot() diff --git a/docs/index.md b/docs/index.md index 16d778b69e..f88054ae8b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -90,6 +90,7 @@ structure of our package and hosting setup may be interested in the following se :maxdepth: 1 architecture +dataset-lifecycle maintenance Source repository (GitHub) ``` diff --git a/docs/requirements.txt b/docs/requirements.txt index 2146ad7fd7..0b008834f7 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -4,3 +4,4 @@ myst-parser >=0.18.1 sphinx-copybutton >=0.5.0 pygments-graphql-lexer >=0.1.0 sphinxext-rediraffe >=0.2.7 +sphinxcontrib-mermaid >=2.0.1 diff --git a/docs/specs/2026-04-06-doi-lifecycle-design.md b/docs/specs/2026-04-06-doi-lifecycle-design.md deleted file mode 100644 index 75af34bb28..0000000000 --- a/docs/specs/2026-04-06-doi-lifecycle-design.md +++ /dev/null @@ -1,221 +0,0 @@ -# DOI Lifecycle State Management — Design Spec - -**Date**: 2026-04-06 -**Status**: Drafted - -## 1. Problem Statement - -OpenNeuro mints DOIs for every snapshot via DataCite's legacy MDS API. The current implementation has three problems: - -1. **DOIs are immediately findable.** A snapshot on an embargoed (private) dataset gets a DOI that is discoverable in DataCite search, leaking the existence of unreleased data. - -2. **DOI state is never updated after creation.** Publishing, deprecating, re-embargoing, or deleting a dataset has no effect on its DOIs at DataCite. A deleted dataset's DOIs remain findable indefinitely. - -3. **Deletion is unnecessarily restrictive.** Any dataset with snapshots requires admin intervention to delete, even when all DOIs are still in draft state and can be cleanly removed from DataCite. - -The `Doi` model (`packages/openneuro-server/src/models/doi.ts`) stores `{ datasetId, snapshotId, doi }` with no state tracking. The DOI creation code (`packages/openneuro-server/src/libs/doi/index.ts`) uses the MDS API, which does not support state transitions. - -## 2. Design Goals - -- **Align DOI discoverability with dataset visibility.** Embargoed datasets get draft DOIs; public datasets get findable DOIs; deprecated snapshots get registered (resolvable but not discoverable) DOIs. -- **Fail fast on metadata issues.** Create DOIs in draft state at snapshot time so invalid metadata is caught before the git tag is created. -- **Reduce admin burden.** Let owners delete their own embargoed datasets when all DOIs are still draft. -- **Migrate existing DOIs.** Backfill state for all existing DOI records and reconcile with DataCite. -- **Handle DataCite failures gracefully.** Detect and recover from state divergence between the local database and DataCite. - -## 3. Architecture - -### System components - -DOI state management spans three layers: - -| Layer | Component | Role | -| ----------------- | ------------------------------------------------------------------- | -------------------------------------- | -| GraphQL resolvers | `packages/openneuro-server/src/graphql/resolvers/publish.ts` et al. | Entry points for lifecycle mutations | -| Dataset service | `packages/openneuro-server/src/datalad/dataset.ts` | Orchestrates dataset state changes | -| DOI service | `packages/openneuro-server/src/libs/doi/index.ts` | Communicates with DataCite API | -| Snapshot service | `packages/openneuro-server/src/datalad/snapshots.ts` | Creates snapshots and mints draft DOIs | -| DOI model | `packages/openneuro-server/src/models/doi.ts` | MongoDB persistence for DOI records | -| Datalad service | `services/datalad/datalad_service/tasks/publish.py` | S3/GitHub export on publish | -| Datalad snapshots | `services/datalad/datalad_service/tasks/snapshots.py` | Git tag creation | - -### Key change: MDS API to REST API - -The current DOI code uses DataCite's MDS API (XML metadata, `PUT /doi`). DOI state transitions (`publish`, `register`, `hide`) require the REST API (`POST/PUT https://api.datacite.org/dois` with JSON payloads). This is the most significant infrastructure change. - -### Data flow - -DOI side effects are injected into existing mutation handlers. No new mutations are added. The DOI service gains three new operations: `publishDoi()`, `hideDoi()`, and `deleteDoi()`, corresponding to DataCite REST API events. - -## 4. DOI State Model - -DataCite defines three DOI states: - -| DOI State | Resolvable | Discoverable | Deletable | When used | -| -------------- | ---------- | ------------ | --------- | --------------------------------------------- | -| **Draft** | No | No | Yes | Embargoed dataset (any snapshot status) | -| **Registered** | Yes | No | No | Public + deprecated snapshot, or re-embargoed | -| **Findable** | Yes | Yes | No | Public dataset, active snapshot | - -### State mapping: dataset lifecycle x snapshot status to DOI state - -| Dataset lifecycle | Snapshot status | DOI state | -| ------------------------ | --------------- | ---------- | -| Embargoed | Active | Draft | -| Embargoed | Deprecated | Draft | -| Public | Active | Findable | -| Public | Deprecated | Registered | -| Embargoed (was Public) | Active | Registered | -| Embargoed (was Public) | Deprecated | Registered | -| Deleted (all DOIs draft) | -- | Deleted | -| Deleted (any non-draft) | -- | Registered | - -### State diagram - -```mermaid -stateDiagram-v2 - [*] --> Draft : createSnapshot() - Draft --> Draft : deprecateSnapshot() / undoDeprecateSnapshot() - Draft --> Findable : publishDataset() - Draft --> [*] : deleteDataset() - Registered --> Findable : publishDataset() / undoDeprecateSnapshot() - Findable --> Registered : updatePublic(false) [admin] / deprecateSnapshot() / deleteDataset() -``` - -### Transition table - -| Trigger | DataCite event | DOI transition | -| ---------------------------------------- | -------------- | ---------------------------- | -| `publishDataset()` | `publish` | Draft/Registered -> Findable | -| `updatePublic(false)` | `hide` | Findable -> Registered | -| `deprecateSnapshot()` (if Public) | `hide` | Findable -> Registered | -| `deprecateSnapshot()` (if Embargoed) | -- | stays Draft | -| `undoDeprecateSnapshot()` (if Public) | `publish` | Registered -> Findable | -| `undoDeprecateSnapshot()` (if Embargoed) | -- | stays Draft | -| `deleteDataset()` (draft DOIs) | DELETE | Draft -> Deleted | -| `deleteDataset()` (findable DOIs) | `hide` | Findable -> Registered | -| `deleteDataset()` (registered DOIs) | -- | stays Registered | - -## 5. Mutation Side Effects - -For each GraphQL mutation that now requires DOI side effects: - -| Mutation | Current behavior | New DOI side effect | Scope | -| ----------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------- | -| `createSnapshot` | Mints DOI via MDS API (findable) | Create DOI in **draft** via REST API. If dataset is already public, immediately transition to **findable**. | Single DOI | -| `publishDataset` | Sets `public=true`, exports to S3 | Call `publish` on all snapshot DOIs. Update local state to **findable**. | All DOIs | -| `updatePublic(false)` | Sets `public=false` | Call `hide` on all snapshot DOIs. Update local state to **registered**. | All DOIs | -| `deprecateSnapshot` | Creates DeprecatedSnapshot doc | If dataset is public: call `hide` on this snapshot's DOI, set state to **registered**. If embargoed: no-op. | Single DOI | -| `undoDeprecateSnapshot` | Removes DeprecatedSnapshot doc | If dataset is public: call `publish` on this snapshot's DOI, set state to **findable**. If embargoed: no-op. | Single DOI | -| `deleteDataset` | Removes dataset | For draft DOIs: DELETE at DataCite. For findable DOIs: `hide`. For registered DOIs: no-op. | All DOIs | -| `createDataset` | Creates empty dataset | No DOI side effect. | -- | - -Side effects execute **after** the primary mutation succeeds but **within the same logical operation**. A DataCite failure on `createSnapshot` is fatal (snapshot creation rolls back). DataCite failures on other mutations are logged and queued for retry (see section 8). - -## 6. Deletion Permissions - -The new rule: dataset owners can delete if all associated DOIs are in draft state. Admin is required when any DOI has left draft. - -| Dataset state | DOI state of snapshots | Who can delete | DOI action at DataCite | -| ------------- | ----------------------- | -------------- | ----------------------------- | -| Draft | No snapshots exist | Owner | N/A | -| Embargoed | All DOIs in draft | Owner | DELETE all DOIs | -| Any | Any registered/findable | Admin only | `hide` findable -> registered | - -**Implementation**: The `deleteDataset()` function in `packages/openneuro-server/src/datalad/dataset.ts` currently checks for admin permissions when snapshots exist. This check changes to: query the `Doi` collection for any records with `state != "draft"` for the dataset. If none found, owner permission suffices. If any found, require admin. - -## 7. Migration - -### Scope - -All existing `Doi` records lack a `state` field. Based on the legacy MDS API behavior, all existing DOIs are expected to be in **findable** state at DataCite. - -### Migration steps - -1. **Add `state` field to Doi model** with no default (allows distinguishing migrated from unmigrated records). -2. **Query DataCite REST API** for each existing DOI to confirm its actual state. -3. **Update local records** with the state reported by DataCite. -4. **Flag discrepancies** -- any DOI not found at DataCite or in an unexpected state is logged for manual review. -5. **For deleted datasets** that still have DOI records: if the DOI is findable at DataCite, transition it to registered via `hide`. - -### Execution - -- Run as a one-time script against the production database. -- Dry-run mode first: report what would change without writing. -- Rate-limit DataCite API calls to avoid hitting quotas. -- Expected volume: one API call per existing DOI record. The DataCite REST API supports bulk lookup but individual GETs are simpler to implement and debug. - -### Rollback - -The migration only adds data (the `state` field). Rolling back means removing the field. No existing data is modified or deleted. - -## 8. Error Handling - -### DataCite unavailable - -| Scenario | Behavior | -| ------------------------------------------ | ------------------------------------------------------------------------ | -| DataCite down during `createSnapshot` | **Fatal.** Snapshot creation fails. No git tag created. User sees error. | -| DataCite down during `publishDataset` | **Non-fatal.** Dataset publishes. DOI transition queued for retry. | -| DataCite down during `deleteDataset` | **Non-fatal.** Dataset deleted locally. DOI cleanup queued for retry. | -| DataCite down during deprecate/undeprecate | **Non-fatal.** Local state updated. DOI transition queued for retry. | - -Rationale: `createSnapshot` is fatal because the DOI string must be embedded in `dataset_description.json` before the git tag. All other mutations have a primary effect (visibility change, deletion) that should not be blocked by a DOI service outage. - -### Metadata validation failure - -DataCite rejects DOI creation if required metadata is missing (e.g., no creators). This fails the `createSnapshot` call with a descriptive error. The user must fix metadata before retrying. - -### Local/remote state divergence - -A reconciliation job should run periodically (e.g., daily cron) to: - -1. Query all local `Doi` records. -2. Compare local `state` with DataCite's reported state. -3. Log any mismatches. -4. Optionally auto-correct by issuing the appropriate DataCite event to match the expected state derived from the dataset lifecycle + snapshot status mapping. - -This handles edge cases like a `hide` call that succeeded at DataCite but failed to update MongoDB, or manual DataCite console changes. - -### Retry strategy - -Failed non-fatal DOI transitions are stored in a retry queue (e.g., a MongoDB collection or Redis list). A background worker retries with exponential backoff, capped at 6 retries over 24 hours. After exhausting retries, the failure is escalated to admin review. - -## 9. Testing Strategy - -### Unit tests - -- **State transition logic**: Given (dataset lifecycle, snapshot status), assert the correct DOI state and DataCite event. Cover all cells in the state mapping table. -- **Deletion permission logic**: Given a set of DOI states for a dataset, assert whether owner or admin permission is required. -- **No DataCite calls for embargoed deprecate/undeprecate**: Verify that no API call is made when the dataset is not public. - -### Integration tests (DataCite test environment) - -DataCite provides a test API at `https://api.test.datacite.org`. Integration tests should: - -- Create a DOI in draft state and verify it is not resolvable. -- Transition draft -> findable via `publish` and verify discoverability. -- Transition findable -> registered via `hide` and verify it resolves but is not discoverable. -- Delete a draft DOI and verify it no longer exists. -- Attempt to delete a registered DOI and verify it fails. -- Submit invalid metadata and verify the error response. - -### Migration dry-run - -- Run the migration script in dry-run mode against a staging database with production data. -- Verify that all DOI records receive a state matching DataCite's reported state. -- Verify that DOIs not found at DataCite are flagged, not silently skipped. - -### End-to-end smoke tests - -- Full lifecycle: create dataset -> snapshot -> publish -> deprecate snapshot -> undeprecate -> re-embargo -> delete. Verify DOI state at each step matches the state mapping table. - -## 10. Open Questions - -The lifecycle document resolved the major design decisions. Remaining implementation-level questions: - -1. **Retry infrastructure**: Use an existing job queue (if one exists in the codebase) or add a new retry mechanism? This is an implementation choice, not a design decision. - -2. **Reconciliation frequency**: Daily reconciliation is suggested, but the appropriate interval depends on operational experience with DataCite reliability. - -3. **`publishDate` bug**: The lifecycle doc notes that `updatePublic(false)` overwrites `publishDate` with the current date rather than clearing it (Gap 4). Should this be fixed as part of this work or tracked separately?