Skip to content

Publish test-count badges through the documentation site #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
timlichtenberg merged 2 commits into from
Jun 1, 2026
Merged

Publish test-count badges through the documentation site #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5e625a7
Publish test-count badges through the documentation site
timlichtenberg Jun 1, 2026
3949a11
Fetch full history in the docs build so setuptools-scm sees the tags
timlichtenberg Jun 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .github/badges/tests-integration.json
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions .github/badges/tests-total.json
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions .github/badges/tests-unit.json
View file
Open in desktop

This file was deleted.

8 changes: 8 additions & 0 deletions .github/workflows/docs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,19 @@ jobs:
steps:
- uses: actions/configure-pages@v5
- uses: actions/checkout@v6
with:
# setuptools-scm derives the version from git tags; the badge step
# installs the package, so the full tag history must be present.
fetch-depth: 0
- uses: actions/setup-python@v6
with:
python-version: 3.x
- run: pip install zensical markdown-include pymdown-extensions mkdocstrings mkdocstrings-python
- run: zensical build --clean
- name: Add test-count badges to the site
run: |
pip install ".[develop]"
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fetch tags before installing the package

On normal main pushes where HEAD is not itself a release tag, the docs workflow's default actions/checkout is a depth-1 checkout without tag history, but this newly added install now builds a package whose version is derived by setuptools-scm (dynamic = ["version"] with no fallback in pyproject.toml). The deleted badge workflow used fetch-depth: 0 before this same install; without restoring that here, pip install ".[develop]" can fail version detection before tools/generate_test_badges.py runs, so the docs/badge deployment remains broken. The setuptools-scm docs also call out using fetch-depth: 0 for GitHub Actions: https://setuptools-scm.readthedocs.io/en/latest/integrations/

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

@timlichtenberg timlichtenberg Jun 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without restoring [fetch-depth: 0] here, pip install ".[develop]" can fail version detection before tools/generate_test_badges.py runs

Added fetch-depth: 0 to the docs checkout, thanks. One correction for the record: the build does not actually fail without it. I tested a depth-1, no-tags checkout, and setuptools-scm falls back to a 0.0.post1.dev1+g<sha> version and the wheel builds fine; since the badge step only runs pytest --collect-only, the counts are unaffected by the version string. The full fetch is still the right call (it yields the real version and matches the setuptools-scm guidance), just not a deploy-breaker.

python tools/generate_test_badges.py --out site/badges/
- uses: actions/upload-pages-artifact@v4
with:
path: site
Expand Down
74 changes: 0 additions & 74 deletions .github/workflows/publish-test-badges.yml
View file
Open in desktop

This file was deleted.

10 changes: 5 additions & 5 deletions docs/Explanations/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![codecov](https://img.shields.io/codecov/c/github/FormingWorlds/CALLIOPE?label=coverage&logo=codecov&color=brightgreen)](https://app.codecov.io/gh/FormingWorlds/CALLIOPE)
[![Unit Tests](https://img.shields.io/github/actions/workflow/status/FormingWorlds/CALLIOPE/tests.yaml?branch=main&label=Unit%20Tests&color=brightgreen)](https://github.com/FormingWorlds/CALLIOPE/actions/workflows/tests.yaml)
[![Integration Tests](https://img.shields.io/github/actions/workflow/status/FormingWorlds/CALLIOPE/nightly.yml?branch=main&label=Integration%20Tests&color=brightgreen)](https://github.com/FormingWorlds/CALLIOPE/actions/workflows/nightly.yml)
[![tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/FormingWorlds/CALLIOPE/main/.github/badges/tests-total.json)](https://proteus-framework.org/testing)
[![tests](https://img.shields.io/endpoint?url=https://proteus-framework.org/CALLIOPE/badges/tests-total.json)](https://proteus-framework.org/testing)

Tests verify that the code does what was written; physical correctness is judged by data, not by tests.
The suite catches regressions in equilibrium chemistry, solubility laws, oxygen-fugacity buffers, and the hybrid solver, but it cannot certify that those formulae match nature.
Expand Down Expand Up @@ -180,13 +180,13 @@ python tools/check_test_quality.py --physics-invariant-status
## Public-facing badges versus internal taxonomy

Public-facing badges (README, project website) collapse `smoke + integration + slow` into a single `Integration Tests` category, because a four-way taxonomy is confusing to non-developer readers.
The four-marker internal scheme remains for CI infrastructure granularity: the PR gate runs `(unit or smoke)`, the nightly runs everything, and the test-count badge fetches the JSON files written by the publish-test-badges workflow.
The four-marker internal scheme remains for CI infrastructure granularity: the PR gate runs `(unit or smoke)`, the nightly runs everything, and the test-count badge fetches the JSON files written into the documentation site during the docs deploy.

## Badge system

Three JSON files at `.github/badges/tests-{total,unit,integration}.json` are rewritten by `.github/workflows/publish-test-badges.yml` on every push to `main` (paths-filtered to source, tests, tools, and `pyproject.toml`).
Shields.io fetches them live via the endpoint URL embedded in the test-count badge.
The publish workflow auto-commits the badges with `[skip ci]` and retries the push up to three times to absorb concurrent main-branch updates.
The documentation deploy (`.github/workflows/docs.yaml`) regenerates three JSON files, `tests-{total,unit,integration}.json`, from `pytest --collect-only` and writes them into the published site under `badges/`.
Shields.io fetches them live from the site via the endpoint URL embedded in the test-count badge.
The counts refresh on every documentation build, so they track the suite without running it.

## Coverage gates

Expand Down
2 changes: 1 addition & 1 deletion docs/How-to/build_tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![codecov](https://img.shields.io/codecov/c/github/FormingWorlds/CALLIOPE?label=coverage&logo=codecov)](https://app.codecov.io/gh/FormingWorlds/CALLIOPE)
[![Unit Tests](https://img.shields.io/github/actions/workflow/status/FormingWorlds/CALLIOPE/tests.yaml?branch=main&label=Unit%20Tests)](https://github.com/FormingWorlds/CALLIOPE/actions/workflows/tests.yaml)
[![Integration Tests](https://img.shields.io/github/actions/workflow/status/FormingWorlds/CALLIOPE/nightly.yml?branch=main&label=Integration%20Tests)](https://github.com/FormingWorlds/CALLIOPE/actions/workflows/nightly.yml)
[![tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/FormingWorlds/CALLIOPE/main/.github/badges/tests-total.json)](https://proteus-framework.org/testing)
[![tests](https://img.shields.io/endpoint?url=https://proteus-framework.org/CALLIOPE/badges/tests-total.json)](https://proteus-framework.org/testing)

This page is the practical contributor guide for adding or modifying a test in CALLIOPE.
The conceptual framing (marker scheme, badge system, coverage gates, AST linter) lives in the [testing suite](../Explanations/testing.md) explainer; read it first if you have not yet.
Expand Down
Loading