Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@ jobs:
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-report
path: implementations/python/coverage.xml
path: |
implementations/python/coverage.xml
implementations/python/coverage.json

fuzz:
runs-on: ubuntu-latest
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ htmlcov/
.cache
nosetests.xml
coverage.xml
coverage.json
*.cover
*.py.cover
.hypothesis/
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: policy

policy:
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s policy
uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s policy $(if $(strip $(RAES_REQUIREMENT_UID)),,-- --skip-requirement)
1 change: 1 addition & 0 deletions docs/decisions/adrs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,3 +251,4 @@ adr-101-adversarial-participant-flow-control
| [100](adr-100-participant-crossing-bisimulation.md) | Proof-Bearing Participant-Crossing Bisimulation | accepted | 2026-07-29 |
| [101](adr-101-adversarial-participant-flow-control.md) | Adversarial Participant Boundary Flow Control | accepted | 2026-07-30 |
| [102](adr-102-mixed-cross-backend-participant-control.md) | Mixed Cross-Backend Participant Control | accepted | 2026-07-31 |
| [103](adr-103-branch-aware-python-coverage-policy.md) | Branch-Aware Python Coverage Policy | accepted | 2026-08-13 |
105 changes: 105 additions & 0 deletions docs/decisions/adrs/adr-103-branch-aware-python-coverage-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# ADR-103: Branch-Aware Python Coverage Policy

## Status

accepted

## Date

2026-08-13

## Classification

Classification: FM0
Required artifacts: repository configuration and focused policy tests
Waivers: none

## Context

ADR-014 makes nox the canonical verification graph and already places Python
coverage collection in that graph. The coverage configuration measures
statements only, omits some repository-owned Python entry points, and accepts a
50% aggregate floor even though the integrated repository is close to 90% line
coverage. Branch outcomes are therefore invisible and the threshold does not
represent the repository's quality policy.

Coverage is useful as a regression guard and review signal, but it is not a
proof of behavioral correctness. Requiring 100% aggregate or changed-code
coverage creates incentives to execute lines without asserting behavior,
over-mock difficult boundaries, distort defensive code, or add exclusions that
serve the metric rather than the design.

ADR-015 also defines the trust model for repository-authored policy: gates
prevent accidental regressions, while review protects the mutable gate itself.
An in-repository coverage checker cannot provide an adversarial security
boundary because the checker, its configuration, tests, and invocation can all
change in the same pull request.

## Decision

1. Coverage.py remains the authority for Python statement, exclusion, and
branch semantics. The repository does not maintain a second Python semantic
analyzer for coverage.
2. Canonical unit and integration coverage is combined before reporting.
Coverage collection includes shipped packages, repository tooling,
`noxfile.py`, and the Hatch build hook. Tests, documentation, generated
environments, and acquired caches remain outside the production denominator.
3. Canonical verification enforces a fixed 90% aggregate **line-coverage**
floor. The floor is an explicit quality threshold, not a manually advanced
ratchet. Falling below it fails verification; exceeding it does not require a
metadata update.
4. Canonical verification measures branch coverage and publishes it in XML and
JSON reports. This ADR establishes no aggregate or changed-branch threshold;
branch results remain visible for review until repository evidence justifies
a separate threshold.
5. The repository does not impose blanket 100% changed-line or changed-branch
coverage. Reviewers apply the proportionate FM0-FM3 assurance policy from
ADR-007 and ADR-018 when a change needs stronger evidence than the aggregate
floor.
6. Coverage configuration and report handling use ordinary fail-closed input
checks for malformed or missing data. They do not attempt to resist a pull
request author who can edit the gate itself; that would require a separately
accepted immutable-CI trust boundary.
7. Coverage.py XML remains the SonarCloud input. JSON is published alongside it
so the line-only threshold and branch totals are inspectable without
reinterpreting Python source.

## Alternatives Considered

### A manually maintained aggregate ratchet

Rejected. A floor that can only be raised still permits all gains above a stale
recorded value to be lost. It depends on recurring manual updates while
presenting itself as automatic regression protection.

### 100% coverage for changed code

Rejected. Diff boundaries are a poor proxy for behavioral risk, particularly
for small changes, defensive paths, multiline statements, and branch exits.
Enforcing the rule also requires semantic diff machinery disproportionate to
the repository's accidental-regression trust model.

### Exact base-versus-head aggregate comparison

Not adopted. Running the complete unit and integration suite at both revisions
would provide an automatic debt comparison, but it would approximately double
the coverage portion of canonical verification. The fixed floor and branch
visibility provide the intended guardrail at materially lower cost.

### SonarCloud as the only coverage gate

Rejected. SonarCloud provides useful new-code line and condition metrics, but it
is an external service and is unavailable to some pull-request contexts. The
canonical local graph must retain its repository-owned aggregate floor.

## Consequences

- Coverage reports include branch information and all agreed repository-owned
Python surfaces.
- The coverage policy is understandable from standard Coverage.py behavior and
a small line-total check rather than a custom semantic analyzer.
- The 90% floor remains stable and meaningful without maintenance commits.
- Branch coverage may regress without independently failing while no branch
threshold is adopted; the reports and SonarCloud make that movement visible.
- Changes that intentionally alter measured scope must update this ADR or be
justified by a later superseding decision.
3 changes: 3 additions & 0 deletions docs/decisions/adrs/adr-index.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -500,3 +500,6 @@ adrs:
- id: ADR-102
path: docs/decisions/adrs/adr-102-mixed-cross-backend-participant-control.md
pin: cfcb0f5b7291b47fe6ff6096bb18dd9063b83ee5d4f7f6a9cafbdd06f21bf716
- id: ADR-103
path: docs/decisions/adrs/adr-103-branch-aware-python-coverage-policy.md
pin: a44acbc2db1b5349ba316ba0d09bd9b46310db016cb87937b08bc6bc107755aa
28 changes: 9 additions & 19 deletions implementations/python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -123,29 +123,19 @@ addopts = "-m 'not fuzz and not integration and not docker'"
timeout = 120

[tool.coverage.run]
source = [
"raes_contracts",
"raes_backend_protocols",
"raes_backend_stubs",
"raes_backend_libvirt",
"raes_operations",
"raes_reference_backend",
"raes_cli",
"raes_conformance",
"raes_mcp",
"raes_processor",
"raes_runtime",
"raes",
# Repo CI tooling (issue #54): the `tools/` tree lives at the repo root and
# is imported as the top-level `tools` package via pytest's `pythonpath`
# (`../..`). Coverage matches source packages by module name, so its
# `tools.*` modules are measured even though pytest runs from
# `implementations/python/`.
"tools",
branch = true
relative_files = false
source = ["../.."]
omit = [
"*/.cache/*",
"*/docs/*",
"*/implementations/python/.venv/*",
"*/implementations/python/tests/*",
]

[tool.coverage.report]
show_missing = true
include_namespace_packages = true
exclude_also = [
# The reference backend's OCI subprocess leaf (RUN-314): real container
# IO, exercised only by the opt-in docker integration tests, never in the
Expand Down
86 changes: 82 additions & 4 deletions implementations/python/tests/test_repo_policy_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ def chdir(self, _path: Path):
assert kwargs["env"] == {"COVERAGE_FILE": str(coverage_file)}

session.commands.clear()
monkeypatch.setattr(noxfile, "_enforce_line_coverage", lambda _path: 90.0)
noxfile._run_pytest(
session,
"-m",
Expand All @@ -143,8 +144,8 @@ def chdir(self, _path: Path):
for command, options in session.commands
if command[:4] == ("uv", "run", "--frozen", "coverage")
]
assert [command[4] for command, _options in coverage_commands] == ["xml", "report"]
assert coverage_commands[-1][0][-2:] == ("--fail-under=50", "--format=total")
assert [command[4] for command, _options in coverage_commands] == ["xml", "json", "report"]
assert coverage_commands[-1][0][-1:] == ("--format=total",)
assert all(options["env"] == {"COVERAGE_FILE": str(coverage_file)} for _, options in coverage_commands)


Expand Down Expand Up @@ -299,21 +300,98 @@ def chdir(self, _path: Path):
return nullcontext()

session = FakeSession()
monkeypatch.setattr(noxfile, "_enforce_line_coverage", lambda _path: 90.0)
noxfile._finalize_parallel_coverage(session, tmp_path)

coverage_commands = [
(command, options)
for command, options in session.commands
if command[:4] == ("uv", "run", "--frozen", "coverage")
]
assert [command[4] for command, _options in coverage_commands] == ["combine", "xml", "report"]
assert [command[4] for command, _options in coverage_commands] == ["combine", "xml", "json", "report"]
assert coverage_commands[0][0][5:] == ("--keep", str(tmp_path))
assert coverage_commands[-1][0][-2:] == ("--fail-under=50", "--format=total")
assert coverage_commands[-1][0][-1:] == ("--format=total",)
assert all(
options["env"] == {"COVERAGE_FILE": str(tmp_path / ".coverage")} for _command, options in coverage_commands
)


def test_coverage_configuration_measures_branches_and_repository_python() -> None:
config = tomllib.loads((REPO_ROOT / "implementations" / "python" / "pyproject.toml").read_text(encoding="utf-8"))
run = config["tool"]["coverage"]["run"]
report = config["tool"]["coverage"]["report"]

assert run["branch"] is True
assert run["source"] == ["../.."]
assert set(run["omit"]) == {
"*/.cache/*",
"*/docs/*",
"*/implementations/python/.venv/*",
"*/implementations/python/tests/*",
}
assert report["include_namespace_packages"] is True


def test_line_coverage_threshold_is_fixed_at_ninety_percent(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
noxfile = load_noxfile_with_fake_nox(monkeypatch)
report_path = tmp_path / "coverage.json"
report_path.write_text(
json.dumps({"totals": {"covered_lines": 90, "num_statements": 100}}),
encoding="utf-8",
)

assert noxfile._enforce_line_coverage(report_path) == 90.0

report_path.write_text(
json.dumps({"totals": {"covered_lines": 899, "num_statements": 1000}}),
encoding="utf-8",
)
with pytest.raises(RuntimeError, match="89.900% is below required 90.000%"):
noxfile._enforce_line_coverage(report_path)


def test_make_policy_skips_only_requirement_governance_without_a_uid() -> None:
environment = os.environ.copy()
environment.pop("RAES_REQUIREMENT_UID", None)
requirement_free = subprocess.run(
("make", "--dry-run", "policy"),
cwd=REPO_ROOT,
env=environment,
check=True,
capture_output=True,
text=True,
).stdout
environment["RAES_REQUIREMENT_UID"] = "ASR-505"
requirement_scoped = subprocess.run(
("make", "--dry-run", "policy"),
cwd=REPO_ROOT,
env=environment,
check=True,
capture_output=True,
text=True,
).stdout

assert requirement_free.rstrip().endswith("-- --skip-requirement")
assert "--skip-requirement" not in requirement_scoped


def test_hook_policy_context_skips_only_requirement_free_branches(monkeypatch: pytest.MonkeyPatch) -> None:
noxfile = load_noxfile_with_fake_nox(monkeypatch)
monkeypatch.delenv("RAES_REQUIREMENT_UID", raising=False)
monkeypatch.setattr(noxfile, "_git_lines", lambda *_args: ["1104-minimal-coverage-policy"])
assert noxfile._requirement_aware_policy_args("--staged") == ["--staged", "--skip-requirement"]

monkeypatch.setattr(noxfile, "_git_lines", lambda *_args: ["1104-ASR-505-coverage-policy"])
assert noxfile._requirement_aware_policy_args("--staged") == ["--staged"]

monkeypatch.setenv("RAES_REQUIREMENT_UID", "ASR-505")
monkeypatch.setattr(noxfile, "_git_lines", lambda *_args: ["1104-minimal-coverage-policy"])
assert noxfile._requirement_aware_policy_args("--staged") == ["--staged"]


def test_docs_graph_uses_curated_root_and_reader_style_gate(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
Expand Down
Loading
Loading