Skip to content

feat(tests): add pytest-codeblocks for documentation code snippet CI#198

Open
haoyu-haoyu wants to merge 3 commits intodotimplement:mainfrom
haoyu-haoyu:feat/doc-code-ci
Open

feat(tests): add pytest-codeblocks for documentation code snippet CI#198
haoyu-haoyu wants to merge 3 commits intodotimplement:mainfrom
haoyu-haoyu:feat/doc-code-ci

Conversation

@haoyu-haoyu
Copy link
Copy Markdown

Summary

Add automated testing infrastructure for documentation code examples using pytest-codeblocks, so code snippets stay in sync with the evolving codebase.

Changes

New dependency

  • pytest-codeblocks>=0.17.0,<0.18 added to dev dependencies

CI integration

  • New step in .github/workflows/ci.yml runs pytest --codeblocks on docs/quickstart.md, docs/cookbook/, and docs/tutorials/
  • Runs on Python 3.12 only (pytest-codeblocks not verified for 3.13)
  • Uses continue-on-error: true so snippets can be incrementally enabled

Skip markers

  • 56 Python code blocks across 11 doc files marked with <!--pytest.mark.skip--> (these require healthchain package + external services)
  • Files covered: quickstart, all 5 cookbooks, 6 tutorial pages

pytest configuration

  • Added [tool.pytest.ini_options] with codeblocks marker in pyproject.toml

Usage

# Test all doc snippets
uv run pytest --codeblocks docs/quickstart.md docs/cookbook/ docs/tutorials/

# Test a specific file
uv run pytest --codeblocks docs/quickstart.md

To make a snippet testable, remove its <!--pytest.mark.skip--> marker and ensure it runs standalone.

Design decisions

  • Skip-first approach: All existing snippets are skipped initially since they depend on the healthchain package, FHIR servers, spaCy models, etc. Snippets can be incrementally enabled as they are made self-contained.
  • Scoped to curated docs: Only quickstart, cookbook, and tutorials are tested — reference/API docs are excluded to avoid executing auto-generated content.
  • Single Python version: Runs on 3.12 only to avoid compatibility issues with the pytest-codeblocks package.

Closes #164

Add automated testing infrastructure for documentation code examples
using pytest-codeblocks:

- Add pytest-codeblocks>=0.17.0 to dev dependencies
- Add [tool.pytest.ini_options] with codeblocks marker
- Add doc snippet test step to CI workflow
- Mark 56 Python code blocks across 11 doc files with skip markers
  (these require healthchain package + external services to run)

The CI step runs with continue-on-error initially so snippets can be
incrementally enabled as they are made self-contained.

Usage:
  uv run pytest --codeblocks docs/   # Test all doc snippets
  uv run pytest --codeblocks docs/quickstart.md  # Test specific file

To make a snippet testable, remove its <!-- pytest-codeblocks:skip -->
marker and ensure it runs standalone (no external dependencies).

Closes dotimplement#164
- Fix skip marker format: replace legacy `pytest-codeblocks:skip`
  with current `pytest.mark.skip` syntax (56 markers updated)
- Restrict CI doc test to Python 3.12 only (pytest-codeblocks not
  verified for 3.13)
- Narrow CI scope to quickstart + cookbook + tutorials (avoids
  executing shell blocks in reference docs)
- Update pyproject.toml comment with correct skip syntax
- Add skip markers to 3 missed cookbook files (format_conversion,
  index, setup_fhir_sandboxes) — 10 Python blocks
- Add skip markers to all bash/sh/shell blocks in scoped docs
  to prevent pytest-codeblocks from executing install/setup scripts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CI for documentation code snippets

1 participant