GREI 2.1.2: Dataverse Y5 monitor metadata sync with DataCite

[jggautier]

Overview

Y5 2.1 Task: GREI repositories strive to implement components of GREI Metadata Recommendation (version 2)

• 2.1.2 Deliverable: Individual repositories ensure any updates to repository metadata records (e.g., ROR IDs, dates, ORCIDs) are regularly synced in the DOI records hosted at DataCite
• Note: The issue was renamed for GREI Year 5, from "Ensure that we're able to keep in sync the repository's metadata, the metadata in the "Datacite" export, and the metadata we expect DataCite to have" to "GREI 2.1.2: Dataverse Y5 monitor metadata sync with DataCite"

Background

This GitHub issue tracks progress towards the goal of making sure that we're able to keep in sync the metadata in Harvard Dataverse; the metadata included in each dataset's and file's "Datacite" export; and the metadata that we expect DataCite to have, now and when we make changes to the metadata we need to send to DataCite.

Ensuring this will be more important as we make changes to Dataverse that help users describe their data (e.g. IQSS/dataverse-pm#127), and as other folks, like NIH GREI groups, progress plans to evaluate those changes by looking at the metadata that DataCite has in order to measure the quality of the repository's dataset metadata (e.g. IQSS/dataverse-pm#420, IQSS/dataverse-pm#411, IQSS/dataverse-pm#419).

This GitHub issue replaces the narrower GitHub issue at #146, which has conversations and examples about how those three sources of metadata are not in sync, and conversations about potential solutions for syncing those three sources.

This GitHub issue is a followup to IQSS/dataverse#5144, which was closed because we include in the release notes of each Dataverse software version instructions for sending updated metadata to DataCite. We've found that for the folks running several Dataverse installations, this wasn't enough.

For Harvard Dataverse in particular, keeping these metadata sources in sync has been too resource intensive. But work that has been done over the years may make it easier or less resource intensive to do this reliably.

Participants

• Leads: @jggautier and @landreev

Tasks

• @jggautier to followup with Leonid to ask who could do this work when it's prioritized
• Look for cases where DataCite doesn't have the metadata we expect and cases where the "Datacite" export doesn't have the metadata we expect.
  • Try the database query ideas shared later in this GitHub issue. See Google Sheet with database query, list of datasets it produces, and notes about missing metadata and broken DOIs.
  • Review known analyses of Harvard Dataverse metadata that DataCite has, such as Ted Haberman's Google Collab notebook done as part of the NIH GREI's Metadata Game Changers task group
• Investigate how to sync the current metadata in Harvard Dataverse; the metadata included in each dataset's and file's "Datacite" export; and the metadata that we expect DataCite to have. And investigate how to ensure that these sources are in sync for already-published datasets and files when we make changes to the metadata that the repository sends to DataCite. This investigation might lead to additional tasks.
• Based on the investigation, sync the current metadata in Harvard Dataverse; the metadata included in each dataset's and file's "Datacite" export; and the metadata that we expect DataCite to have. And demonstrate that the metadata in the three sources is synced.

Related

• Resend metadata to PID providers when metadata schema used to register PIDs is modified dataverse#5144
• Affiliations entered in affiliation fields are parenthesized in "Datacite" and Schema.org exports dataverse#9330
• QDR-DataCite Scaling dataverse#11832
• Send to DataCite the relationType metadata of files that have PIDs #146

Required Technical Work

• DataCite implementation: DataCiteRESTfullClient.getMetadata() relies on the MDS API which appears to garble UTF8 characters dataverse#12070

Timeline

• Start: October 2025
• End: To be determined; sometime before the end of FY 2026 (June 30, 2026)

[sbarbosadataverse]

This work is dependent on developer time. Things to do include:

• query to see datasets where the globalidcreatetime is earlier than the latest version release time / last deployment that changed the DataCite format
• query to see datasets where the lastexporttime is earlier than the latest version release time /last deployment that changed any export
• try running the reexportAll and pid update commands to see if they scale well enough to use at Harvard (they were not scalable enough in the past, but may be now)

[jggautier]

Status: November 2025

• @jggautier prioritized this work to start in this quarter (Oct. 2025 - Dec. 2025) after hearing from @cmbz and @sbarbosadataverse about other work that I'll start in 2026 instead. He updated the "Tasks" list in this issue's first comment about looking for cases where DataCite doesn't have the metadata we expect and cases where the "Datacite" export doesn't have the metadata we expect, describing those cases in this GitHub issue, and trying the database queries suggested in a later comment.
• @jggautier finished looking for cases where DataCite doesn't have the metadata we expect and cases where the "Datacite" export doesn't have the metadata we expect and trying the database queries suggested in a later comment:
  • @jggautier checked examples of published datasets with unsynced metadata mentioned in other related and closed GitHub issues (#5144, #9330, #146), and checked random older and newer datasets for known past issues: file PIDs not listed, author affiliations wrapped in parentheses, missing version numbers, missing resourceTypeGeneral. He didn't find any cases of unsynced metadata.
  • @jggautier ran Ted Haberman's Google Collab notebook and reviewed the analyses to check for any obvious cases of unsynced metadata for the 66 published datasets whose metadata the notebook evaluated. @jggautier confirmed that cases of unsynced metadata he saw when using the notebook earlier in the year, involving missing version numbers and ResourceTypeGeneral values, have been resolved. DataCite has that metadata now.
  • @jggautier queried the Harvard Dataverse database using the database queries that @qqmyers suggested but limited to published datasets whose latest version is not deaccessioned, added the query results to a Google Sheet, and noted a few cases of unsynced metadata to discuss with the team.
• @jggautier followed up with colleagues about next steps for making sure that metadata is synced.

[jggautier]

I noticed this morning that the DataCite metadata from the DataCite API, like what we see at https://api.datacite.org/dois/10.7910/DVN/ZQU5DA, isn't always the same as the DataCite metadata we can get from the "DataCite XML" and "DataCite JSON" downloads on DataCite Commons, like what we can download from the "Download Metadata" popup at https://commons.datacite.org/doi.org/10.7910/dvn/zqu5da:

For example, for the dataset at https://doi.org/10.7910/DVN/ZQU5DA, the ROR ID for the affiliation of the first author, Jéssica Trigo da Fonseca, is missing from the DataCite API call for that dataset:

The ROR ID for that author's affiliation is present in the "DataCite JSON" and "DataCite XML" downloads on the DataCite Commons page for that dataset:

"DataCite JSON" download

"DataCite XML" download

I noticed this after Jim's Dataverse Google Group post about ROR API changes prompted me to look for any cases where DataCite doesn't have ROR IDs that Harvard Dataverse adds to depositors' dataset metadata, from the external controlled vocabulary feature.

As far as I know, several efforts to measure the completeness of metadata in repositories, like efforts from groups in the NIH GREI, use the https://api.datacite.org/dois/{doi} DataCite API endpoint or other endpoints like it to get metadata for their analysis, so when metadata we expect DataCite to have is missing from what those endpoints return, it can reduce confidence in those measurements and make it harder to make design decisions based on them.

I'm going to ask folks on the dv-metadata-coordination group if they know why this might be happening. If they're not sure I'll reach out to folks at DataCite.

[jggautier]

Ah, false alarm. On DataCite's page about their REST API (https://support.datacite.org/docs/api-get-doi), there's a note about needing to add affiliation=true to the API call so that it includes affiliation identifier details:

So for the examples in my last comment, we would use https://api.datacite.org/dois/10.7910/DVN/ZQU5DA?affiliation=true to see the ROR of the affiliation of the author Jéssica Trigo da Fonseca:

Ted Habermann is one of the folks using DataCite metadata as part of an NIH GREI project to measure the completeness of metadata, and his Python notebook adds that affiliation=true param to its API calls. Other groups working on similar things are on the DataCite team so are probably already aware of this.

[landreev]

OK, looking at some very recently created and published datasets, the output of our "datacite xml" metadata API is 1:1 identical to the encoded xml fragment in the output of the Datacite API.
for ex.,

curl "https://dataverse.harvard.edu/api/datasets/export?exporter=Datacite&persistentId=doi:10.7910/DVN/899DIN" > 899DIN.xml

curl "https://api.datacite.org/dois/10.7910/DVN/899DIN?detail=true" | jq . > 899DIN.json

grep xml 899DIN.json 
      "xml": "PD94bWwgdmVyc2lvbj ... truncated ... 4KPC9yZXNvdXJjZT4=",

echo "PD94bWwgdmVyc2lvbj ... truncated ... 4KPC9yZXNvdXJjZT4=" | base64 -d > 899DIN_decoded.xml

diff 899DIN_decoded.xml 899DIN.xml

I'm going to assume that this is the desired outcome, and we want this to be the case for all of our published datasets.

[landreev]

... repeating the above experiment for doi:10.7910/DVN1/22636 (one of the out-of-sync cases mentioned in the spreadsheet), lots of missing, and some different fields in the Datacite's version of the xml.

[landreev]

I'll be updating the Datacite registration of our datasets as soon as I have Jim's latest mods deployed.

[jggautier]

Status: December 2025

• We patched Harvard Dataverse with changes that improve scaling of requests to DataCite (see PR #11832), taking us a step closer to making it easier to keep in sync the repository's metadata, the metadata in the "Datacite" export, and the metadata we expect DataCite to have