re3data-policy-magpie 🐦‍⬛

A small CLI pipeline that fetches repository metadata from re3data.org, downloads associated policy documents, and cleans them into structured Markdown using an LLM.

What it does

1. Fetch the public re3data repository catalog via the re3data API.
2. Parse per-repository XML to extract metadata, especially policy links.
3. Download policy documents (PDFs kept as binary, web pages converted to text).
4. Clean the downloaded text into well-formatted Markdown with an LLM.

Install

Requires Python >=3.10<3.14

The project uses uv:

git clone https://github.com/UB-Mannheim/re3data-magpie.git
cd re3data-policy-magpie
uv sync
source .venv/bin/activate

Or with pip:

pip install -e .

Quick start

# 1. Fetch the re3data repository list
magpie fetch --to-json

# 2. Enrich repositories with metadata and policy links, build subjects index
magpie parse --limit 10 --delay 1

# 3. Download policy documents (optionally filtered by DFG subject)
magpie download --limit 10 --subjects "Economics,Computer Science"

# 4. Clean documents into Markdown via an LLM
magpie clean --model groq/llama-3.3-70b-versatile --max_tokens 10000

CLI reference

Run magpie --help for the full list of commands.

Command	Purpose	Default output
magpie fetch	Fetch repository list from re3data	./data/api_data/re3data-repos.xml
magpie parse	Query details for each repository, then rebuild the subjects index	./data/api_data/re3data-repos-metadata-dump.json
magpie index	Rebuild subjects_index.list from an existing metadata dump	./data/api_data/subjects_index.list
magpie download	Download policy PDFs / web pages, optionally filtered by DFG subject	./data/policies/raw/
magpie clean	Convert downloaded text/PDF into Markdown with an LLM	./data/policies/processed/clean_markdown/

Filtering downloads by subject

magpie download --subjects "Economics,Computer Science" restricts downloads to repositories tagged with those DFG subjects. Subject names are validated against subjects_index.list (built by parse or index) and unknown terms raise an error with close-match suggestions.

Environment variables

magpie clean reads API credentials from a .env file in the project's root directory. Set the variables matching your LiteLLM provider, for example:

GROQ_API_KEY=...
GROQ_BASE_URL=https://api.groq.com/openai/v1

Use any LiteLLM-compatible model string with the CLI option --model <provider>/<model>.

Output layout

data/
├── api_data/
│   ├── re3data-repos.xml              # Raw re3data catalog
│   ├── re3data-repos-index.json       # id, doi, name, link (fetch --to-json)
│   ├── re3data-repos-metadata-dump.json  # Enriched per-repo metadata
│   └── subjects_index.list            # Unique DFG subject names
├── policies/
│   ├── raw/                   # Downloaded PDFs and .txt files + download_manifest.json
│   └── processed/
│       ├── clean_markdown/    # Final cleaned Markdown files
│       └── failed_markdown/   # Empty/unusable inputs skipped during cleaning

Why this exists

Research data repositories listed in re3data often expose policy documents in inconsistent formats (HTML pages, PDFs, wikis). This tool automates collecting those documents and normalizing them into clean Markdown for downstream analysis, indexing, or annotation.