Force ROR as sole authority on organization identity with disambiguation techniques - #33
Force ROR as sole authority on organization identity with disambiguation techniques#33epistrephein wants to merge 3 commits into
Conversation
An academic publisher was ranking among the most prolific research institutions in the citation counts. The cause was in how we named organizations: OpenAIRE often attaches several ROR identifiers to a single organization record, expressing its own uncertainty about which entity the record describes, and we were simply taking the first one and adopting its name. Where a university and its publishing house both appeared, the university could end up published under the publisher's name. Several hundred organizations were affected this way, including some of the largest in the dataset. Rather than patch the naming step, treat ROR as the authority on what an organization is. Organizations are now identified by their ROR record throughout the pipeline; OpenAIRE identifiers survive only as a means of joining affiliation data and never appear in the output. An organization is published only when it can be resolved to exactly one ROR record, with OpenAIRE's own name and country used solely to choose between competing candidates. Everything a reader sees (name, identifier, country) now comes from ROR and from nowhere else. The cost is coverage. Organizations that cannot be tied to a ROR record are excluded entirely, which removes about a quarter of the rows but only a small fraction of the citations, since the excluded mass is long-tail noise: laboratories, departments and misspelled duplicates of institutions that are themselves counted correctly. No well-curated organization that previously ranked highly was lost. One caveat matters for interpretation. The exclusion does not fall evenly across countries, because some national research systems appear in OpenAIRE as uncurated affiliation strings far more often than others. France in particular loses substantially more than average and Italy substantially less, which is enough to reorder the country rankings. Country totals should not be compared across countries without accounting for this. Organization rankings within a country are largely unchanged, and where names differ from before it is generally because they are now correct.
Citation counts: OLD vs NEW diff overview (AI-made)First 25 rows of every file in The Rows are compared position by position. The marker column reads:
SNSOrganizations — incoming14 identical, 3 recounted, 8 reordered.
Organizations — outgoing15 identical, 1 relabelled, 2 recounted, 7 reordered.
Countries — incoming0 identical, 12 recounted, 13 reordered.
Countries — outgoing0 identical, 12 recounted, 13 reordered.
UNIBOOrganizations — incoming18 identical, 4 relabelled, 3 recounted.
Organizations — outgoing14 identical, 4 relabelled, 3 recounted, 4 reordered.
Countries — incoming0 identical, 11 recounted, 14 reordered.
Countries — outgoing0 identical, 12 recounted, 13 reordered.
UNIMIOrganizations — incoming15 identical, 2 relabelled, 4 recounted, 4 reordered.
Organizations — outgoing16 identical, 5 relabelled, 2 recounted, 2 reordered.
Countries — incoming0 identical, 12 recounted, 13 reordered.
Countries — outgoing0 identical, 14 recounted, 11 reordered.
UNIPDOrganizations — incoming18 identical, 3 relabelled, 2 recounted, 2 reordered.
Organizations — outgoing16 identical, 3 relabelled, 3 recounted, 3 reordered.
Countries — incoming0 identical, 13 recounted, 12 reordered.
Countries — outgoing0 identical, 14 recounted, 11 reordered.
UNITOOrganizations — incoming14 identical, 3 relabelled, 3 recounted, 5 reordered.
Organizations — outgoing17 identical, 5 relabelled, 3 recounted.
Countries — incoming0 identical, 14 recounted, 11 reordered.
Countries — outgoing0 identical, 14 recounted, 11 reordered.
UPOOrganizations — incoming20 identical, 2 recounted, 3 reordered.
Organizations — outgoing20 identical, 2 relabelled, 1 recounted, 2 reordered.
Countries — incoming0 identical, 20 recounted, 5 reordered.
Countries — outgoing0 identical, 18 recounted, 7 reordered.
|
This PR addresses the "Harvard University Press" (and similar entities) matter that affected the released version of the project.
Issue
Harvard University Press (an academic publisher, not a research institution) ranked extremely high among organizations citing or being cited by multiple universities. The cause lay in how organizations were named. OpenAIRE data is very scattered (and often just wrong) and frequently attaches several ROR identifiers to a single organization record, conflating multiple kinda-matching entities that describe the record rather than listing multiple affiliations. Harvard University's record often times carries the identifiers of both the university and its press (example). The original pipeline was designed without awareness of this issue: it assumed singular ROR id and hence took whichever identifier happened to come first and adopted its name, so a lot of Harvard affiliations were published under the publishing house. Other various organizations were mislabelled this way.
Approach and fix
ROR is now the sole authority on organization identity. Organizations are identified by their ROR record throughout the pipeline and OpenAIRE identifiers survive only as a means of joining affiliation data, never reaching the output. An organization is published only when it resolves to exactly one ROR record, with OpenAIRE's own name and country used solely to choose between competing candidates. Every published name, identifier and country therefore comes from ROR and nowhere else and anything that cannot be identified unambiguously is excluded rather than guessed at.
This is the safest approach to avoid other potential issues in OpenAIRE data that have not been discovered yet.
Impact
About a quarter of the rows disappear but only some seven percent of the citations, the excluded mass being long-tail noise: laboratories, departments and misspelled duplicates of institutions already counted correctly. Organization rankings within a country are largely unchanged. The exclusion is not country-neutral, however: France loses far more than average and Italy less, enough to reorder country totals.
Warning
While data general intuitions remained consistent, the data visualizations, website text and article content should be refactored to account for the shifts in position of various actors after the merge of this PR.