iati-docs-management: Docs and tools for developing & maintaining IATI documentation sites

This repo serves two purposes:

1. It hosts a Sphinx site (under docs/) with guidance for maintaining IATI documentation sites.
2. It provides tooling (under scripts/) for managing the estate of IATI documentation repositories as a group.

See also iati-docs-base - the template repo that all IATI docs sites are derived from. iati-docs-base is treated as the authoritative source of truth by the tooling in this repo.

Repository layout

• docs/ - the Sphinx documentation site (maintainer guidance for IATI docs sites).
• scripts/ - reusable tooling for working across all IATI documentation repos.
• example-scripts/ - a parking spot for one-off scripts to keep around for reference after use.

Many of the scripts rely on the gh command-line tool; ensure it is installed and authenticated against the IATI org before running them.

Managing the documentation estate

IATI documentation repos are identified by a GitHub custom repository property named Documentation set to true. The tooling in scripts/ uses this property to discover the estate, so any new docs site must be tagged before it will be picked up.

scripts/repo_manager.py

A Python 3.13 CLI for working across every Documentation-tagged repo, using iati-docs-base as the template. The commands cover three flavours of cross-repo work: template syncs, scripted changes, and manual refactors. Anything that's genuinely a one-off in a single repo should be done by hand on a single branch, not via this tool.

# List every Documentation-tagged repo
python scripts/repo_manager.py list

# Show which repos diverge from the template, with diffs
python scripts/repo_manager.py check

# Sync template files and open PRs against each repo's default branch
python scripts/repo_manager.py sync -m "Sync from template"

# Run an inspection script in each repo (no changes -> no PRs)
python scripts/repo_manager.py run-script ./find-myst-parser.sh

# Run a script that modifies files; PRs are opened automatically
python scripts/repo_manager.py run-script ./bump-sphinx.sh \
    -m "Bump sphinx pin"

# Clone every repo to a persistent dir for hand-edits
python scripts/repo_manager.py checkout-all

# Commit and PR everything you edited under that dir
python scripts/repo_manager.py make-prs \
    --dir /tmp/iati-docs-XXXX -m "Refactor X across the estate"

# Build one repo on main in a fresh venv (sanity check)
python scripts/repo_manager.py build iati-publisher-docs

# Build every repo on main; one-line summary per repo
python scripts/repo_manager.py build-all

# Did my edits break this checkout? Compare against main.
python scripts/repo_manager.py build-compare --dir /tmp/iati-docs-XXXX/iati-publisher-docs

sync

Performs the full template-sync flow per repo: copy template files, commit on a fresh working branch (default iati-docs-management/sync-<timestamp>), push, and open a pull request against the repo's default branch. Repos where every tracked file already matches the template are skipped without producing a PR. The PR review process itself is the human gate - run check first if you'd like to preview the per-file diffs.

It never commits to main - a defence-in-depth check refuses to operate if the working tree somehow ends up on the default branch. Override the branch name with --branch-name and the PR body with --pr-body.

By default, check and sync operate on:

• .readthedocs.yaml
• requirements.in and requirements_dev.in (the canonical pip-compile inputs)
• requirements.txt (tracked but expected to drift)
• .github/workflows/ci.yml
• .vscode/launch.json

For most files, a difference from the template is interesting - it usually means something significant has changed. requirements.txt is the exception: it's regenerated by pip-compile and developers update it locally as part of normal work, so it drifts naturally. check shows it as DIFFERS (expected) and suppresses the diff body, while still reporting genuine problems (a missing file, a missing template). Use --files on either command to override the default list.

run-script

Runs an arbitrary script in each repo. Behaviour is determined by what the script does:

• No filesystem changes anywhere: the run is purely informational. stdout/stderr from each repo is shown; nothing is committed or pushed. Use this for inspection questions like "which repos use this plugin" or "which repos contain this file".
• Filesystem changes in one or more repos: the changes are committed on a fresh working branch (iati-docs-management/script-<script-stem>-<timestamp>) and opened as PRs against each repo's default branch. -m MSG must be provided in this case (the tool will refuse to publish without one) and is used as both the commit message and PR title.
• Non-zero exit, timeout, or invocation error in any repo: the whole run aborts immediately. No further repos are processed and nothing is published, even from repos that succeeded earlier. Investigate the failure before re-running.

The script contract:

• Invoked as <script> <repo-name>.
• cwd is set to the repo's checkout.
• The script must be executable (chmod +x) and have a shebang.
• The script must exit 0 on success - any non-zero exit aborts the entire estate-wide run.
• stdout, stderr, exit code, and the list of changed files are captured and displayed for each repo.
• Per-repo timeout: 5 minutes.

Pass --include-template to also run against iati-docs-base. Even when the script modifies files in the template, those changes are reported but not pushed - the template is treated as the source of truth and is never modified by this tool.

checkout-all + make-prs

For refactors that don't fit a template-sync or a single script - things you need to look at, think about, and edit by hand across the estate - use the two-step manual flow.

checkout-all clones every tagged repo (plus the template) into a fresh /tmp/iati-docs-… directory, creates a working branch (default iati-docs-management/refactor-<timestamp>) in each, and exits without cleanup. The directory persists until macOS reaps /tmp (after roughly three days of inactivity), which is plenty of time to either land PRs or discard the work.

You then edit files in those checkouts however you like - by hand, in your editor, with whatever tools fit the task.

When you're done, make-prs --dir <path> -m "<message>" walks each checkout, stages anything dirty, commits with the supplied message, pushes the working branch, and opens a pull request against the repo's default branch. Repos with nothing to publish (clean tree, no commits ahead of the default) are skipped. The branch name is detected from the first checkout unless --branch-name is given; all checkouts must be on the same branch.

# Step 1: clone everything; note the printed work dir
python scripts/repo_manager.py checkout-all

# Step 2: edit files in the checkouts (by hand, with $EDITOR, etc.)

# Step 3: commit and PR everything you changed
python scripts/repo_manager.py make-prs \
    --dir /tmp/iati-docs-XXXX -m "Drop legacy myst extensions"

The template checkout is included so you can use it as a reference while editing. make-prs ignores it - the template is never published to via this tool.

build, build-all, build-compare

Static checks (check) tell you if files match the template; the build verbs tell you if the docs actually still build. Every build happens in a fresh /tmp clone with a freshly-created .venv-build so the result reflects requirements.txt alone, not whatever's in your working environment.

build <repo> clones one repo at its default branch, installs requirements.txt, runs sphinx-build, and reports the exit code, warning count, and list of HTML pages produced. Use it as a one-off sanity check.

build-all does the same across every Documentation-tagged repo plus the template, with a one-line summary per repo (status, warning count, page count). Use it as an estate-wide health snapshot - especially before/after a template change.

build-compare --dir <path> is the tool for seeing the surface area of a change. It builds the candidate checkout as-is (uncommitted edits included; the working tree is never modified) and a fresh clone of the same repo at --baseline-ref (default main), then reports:

• pages added / removed / modified (with per-page normalised HTML diffs)
• Sphinx warnings added / removed

The report describes what differs; it doesn't adjudicate whether each difference is intended or problematic - that's the operator's call when rolling out an estate-wide change. Exit code is 1 only when the candidate failed to build while the baseline succeeded, so the verb fits into scripts as a build-health gate without falsely failing on intentional content changes. HTML diffs are normalised to strip Sphinx's ?v=<hash> asset cache-busters; extend _HTML_NORMALISERS if a future theme version introduces another build-volatile pattern.

Builds run from the repo's docs/ directory (matching the Makefile / make.bat convention) so any from project_info import ... resolves. Invoking sphinx-build from the repo root instead will fail to import sibling modules; use these verbs (or the per-repo Makefile) and you'll never see that class of problem.

Example - inspection: report which repos use myst_parser. The script always exits 0 and reports findings via stdout; exit codes can't be used to signal "no match" because non-zero would abort the run.

#!/bin/bash
# find-myst-parser.sh - prints how many myst_parser refs each repo has
set -e
if [ -f requirements.txt ] && grep -q "myst_parser" requirements.txt; then
    count=$(grep -c "myst_parser" requirements.txt)
    echo "found $count reference(s)"
else
    echo "not used"
fi

Example - publishing: bump a pinned dependency in requirements.txt:

#!/bin/bash
# bump-sphinx.sh - upgrade sphinx pin
set -e
if [ -f requirements.txt ]; then
    sed -i.bak 's/^sphinx==.*/sphinx==7.4.0/' requirements.txt
    rm requirements.txt.bak
fi

Storage and cleanup

Each invocation clones every tagged repo (plus the template) into a session directory under /tmp that is cleaned up when the command exits. Anything left behind is reaped by the OS's periodic /tmp cleanup.

Python API

For complex checks or syncs that don't fit the CLI, RepoManager exposes run_custom_check and run_custom_sync which accept a callable to run against each checkout. See example_check_python_version and example_sync_gitignore in scripts/repo_manager.py for the expected shape.

Other scripts

• scripts/list_all_docs_repos.sh - print every repo currently tagged Documentation=true.
• scripts/find_untagged_repos.sh - print repos with "docs" in the name that are not tagged. Useful for spotting docs sites that need onboarding into the estate.

Building this site's documentation

"Building" is the process of running Sphinx to turn the source files in docs/ into a navigable HTML site.

There are three ways to build:

• Locally via sphinx-autobuild
• Automatically via ReadTheDocs
• Inside VS Code, using the supplied devcontainer

Using ReadTheDocs

ReadTheDocs builds automatically when a Pull Request is opened, when new commits are pushed to an open PR, and when a PR is merged.

Local live preview

Assuming a Unix-based system:

# Make sure you have python3 venv, e.g. for Ubuntu
# If you're not sure, try creating a venv, and see if it errors
sudo apt-get install python3-venv

# Create and enter a venv
python3 -m venv .ve
source .ve/bin/activate

# Install requirements
pip install -r requirements_dev.txt

# Run sphinx-autobuild
sphinx-autobuild docs docs/_build/html

Then go to http://localhost:8000/ in a browser. Saved changes update the browser automatically. To change the language, edit the language variable in docs/conf.py.

Using VS Code

A .devcontainer/devcontainer.json and .vscode/launch.json are supplied which add sphinx-autobuild as a Run option.

Contributing

Create a branch, make your changes, and open a Pull Request. ReadTheDocs will build a preview so you can see what the site will look like once merged.

Formatting

Python code in this repo is formatted with black. The project is configured to format on save in VS Code; run black . to format manually.

Translations

The process for getting documentation translated is:

• Extract English strings into a .pot file
• Send the .pot file for translation
• Receive .po files from the translation process
• Check the .po files into the repo
• Re-run the build with the translations

Extract strings

cd docs
make gettext
# .pot files are in _build/locale

Send for translation & receive translations

There is no automation for this step. Contact @robredpath for the current process.

Check the files into the repo

Place the files into docs/locale/<lang>/LC_MESSAGES/ (e.g. fr for French).

Re-run the build

On ReadTheDocs, translation projects do not auto-build on Pull Request. To preview a translation, create a Version via the RTD interface and point it at your branch. Translated versions rebuild automatically when the PR is merged.

To build a translation locally:

cd docs
make -e SPHINXOPTS="-D language='fr'" dirhtml

Built docs are in docs/_build/dirhtml.