Skip to content

feat: add coverage quality gates and enforcement - #3

Merged
montge merged 25 commits into
masterfrom
develop
Dec 26, 2025
Merged

feat: add coverage quality gates and enforcement#3
montge merged 25 commits into
masterfrom
develop

Conversation

@montge

@montge montge commented Dec 26, 2025

Copy link
Copy Markdown
Owner

Summary

  • Configure Codecov with enforced thresholds (90% overall, 95% controls, 80% per-component)
  • Add parallel coverage workflow with 3 shards (core, system, tools)
  • Establish baseline coverage (~30%) for improvement tracking

Test plan

  • Verify Codecov thresholds are enforced (PR checks show pass/fail status)
  • Confirm SonarCloud quality gate passes

🤖 Generated with Claude Code

montge and others added 25 commits December 25, 2025 13:08
Add MISRA compliance checking using two complementary tools:
- cppcheck with MISRA C:2012 addon (3,255 baseline findings)
- clang-tidy-automotive for MISRA C:2025/C++:2023 (59,791 findings)

Changes:
- scripts/lint/cppcheck-misra.sh: cppcheck MISRA analysis script
- scripts/lint/clang-tidy-misra.sh: clang-tidy-automotive script
- reports/: comparison and gap analysis documentation
- SConstruct: enable camerad for static analysis on all architectures
- .pre-commit-config.yaml: local quality checks (ruff, mypy, codespell)
- openspec/: spec-driven development configuration
- CLAUDE.md: AI assistant project instructions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Validates OpenSpec changes when openspec/ files are modified.
Uses --strict mode to enforce proper spec formatting.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add pytest-cov to testing dependencies
- Configure coverage.py in pyproject.toml (sources, omits, branch coverage)
- Add scripts/coverage.sh for running coverage locally
- Merge SonarCloud/Codecov workflows from add-sonarcloud branch

Usage: ./scripts/coverage.sh or pytest --cov

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move completed MISRA analysis proposal to archive and update specs.
Task 3.3 (GitHub Actions workflow) remains optional.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update Codecov and SonarCloud workflows to trigger on:
- Push to master and develop
- PRs to master and develop

This supports the gitflow branching model.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 1 of add-code-quality-gates openspec change:
- Establish Python coverage baseline: 33% (target: 90%)
- Document safety-critical modules and their current coverage
- Identify urgent gaps in selfdrived (7%), pandad (8%), controlsd (15%)

Key findings:
- 24,777 Python statements, 9,189 covered
- Safety-critical modules average 50% coverage
- system/ module lowest at 27% coverage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enable coverage threshold enforcement:
- pyproject.toml: fail_under = 33 (baseline)
- codecov.yml: PR coverage requirements (80% for new code)
- Codecov workflow: coverage check step before upload
- scripts/ratchet-coverage.sh: auto-update thresholds as coverage improves

Also:
- .gitignore: add .local/ for temp/AI files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 3 - C++ Coverage:
- SConstruct: add --coverage option for llvm-cov instrumentation
- scripts/cpp-coverage.sh: build, test, and generate coverage reports

Phase 4 - MISRA CI:
- .github/workflows/misra.yml: cppcheck MISRA C:2012 analysis
- Uploads report as artifact, summarizes top findings
- clang-tidy-automotive commented out (requires custom build)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- sonar-project.properties: add C++ coverage path, update exclusions
- Update openspec tasks with completion status

Quality gates now configured:
- Python: fail_under=33 (pyproject.toml)
- Codecov: 80% for new code (codecov.yml)
- SonarCloud: coverage reports configured

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 1 Complete:
- C++ coverage baseline: 67% line, 45% branch (common/, loggerd/)
- MISRA baseline: 3,255 violations (1,003 in non-generated code)

Fixes:
- cpp-coverage.sh: use versioned llvm tools (llvm-profdata-18, llvm-cov-18)
- cpp-coverage.sh: fix test binary paths (test_common, test_logger)
- cpp-coverage.sh: activate venv for scons

New:
- .github/workflows/cpp-coverage.yml: CI workflow for C++ coverage
- reports/misra-baseline.md: MISRA C:2012 baseline analysis
- .gitignore: exclude coverage data files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Critical fixes:
- common/prefix.h: Replace system() calls with std::filesystem::remove_all()
  to prevent command injection via environment variables (CWE-78)

Medium fixes:
- tools/bodyteleop/static/index.html: Add Subresource Integrity (SRI) hashes
  to all CDN scripts, pin versions instead of version ranges (CWE-829)

Low fixes:
- Add explicit minimal permissions to all custom workflows:
  - codecov.yml: contents: read
  - cpp-coverage.yml: contents: read
  - misra.yml: contents: read
  - sonarcloud.yml: contents: read, pull-requests: read

Documentation:
- reports/security-alerts.md: Full analysis of 24 CodeQL alerts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
C++ Coverage:
- Add 65% line coverage threshold check to cpp-coverage.yml
- Fail CI if coverage drops below threshold

MISRA Analysis:
- Add differential checking against 1050 baseline (warning only)
- Exclude additional generated code paths from analysis:
  - selfdrive/controls/lib/*/c_generated_code/
  - selfdrive/locationd/models/generated/
- Separate actionable vs generated code findings in report

Quality Gates Status:
- Python coverage: 33% threshold (enforced)
- C++ coverage: 65% threshold (enforced)
- MISRA: 1050 baseline (warning only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov:
- Fix coverage command: use 'python -m coverage' instead of 'coverage'

C++ Coverage:
- Add missing system dependencies: portaudio19-dev, libcurl4-openssl-dev,
  libjpeg-dev, libusb-1.0-0-dev, libncurses5-dev, opencl headers

SonarCloud:
- Comment out C++ coverage path (handled by separate workflow)
- Prevents NoSuchFileException for cpp-coverage.lcov

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Install --extra testing in uv sync to ensure coverage,
pytest-cov, and other test utilities are available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The scons build parses all SConscript files including cabana,
which requires qmake to be available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add explicit pytest-cov and coverage install (not in frozen lockfile)
- Add ffmpeg dependencies for C++ coverage workflow (loggerd needs libav)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
CI workflows validated:
- MISRA Analysis: SUCCESS
- SonarCloud: SUCCESS
- C++ Coverage: SUCCESS
- Codecov: Running (expected for large test suite)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Split coverage into 3 parallel jobs to complete within timeout:
- Core modules (controls, car, selfdrived, pandad, locationd, monitoring, common)
- System modules
- Tools and other selfdrive modules

Key optimizations:
- Parallel execution reduces wall-clock time from 6h+ to ~90min
- Per-job timeouts (60-90 min) prevent runaway tests
- Per-test timeout reduced to 120s
- Coverage reports merged and uploaded together

This covers ALL modules - just runs them in parallel shards.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The default loadscope scheduler can cause "KeyError: WorkerController"
internal errors when running coverage. Using --dist load provides
simpler distribution without timing-sensitive test collection.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov workflow now working with parallel shards:
- Core, System, Tools jobs run in parallel (~7 min each)
- All 3 coverage reports uploaded to Codecov
- Total wall-clock time reduced from 6+ hours to ~8 minutes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Completed quality gates implementation:
- Python coverage: 33% baseline with fail_under enforcement
- C++ coverage: 67% baseline with 65% threshold
- MISRA analysis: 3,255 violations baselined (1,003 in non-generated)
- Codecov: Parallel shards reduce CI from 6+ hours to ~8 minutes
- SonarCloud: Integrated for quality tracking

Specs updated:
- code-quality: Created with 4 requirements
- misra-analysis: Updated with 1 modified requirement

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Configure Codecov with 90% project / 80% component thresholds
- Enable coverage enforcement in CI workflow
- Add component-level targets (controls 95%, car/selfdrived 90%, etc.)
- Update project.md with coverage requirements and conventions
- Create OpenSpec proposal for tracking quality initiative

Coverage baseline measured at ~30%, establishing target for improvement.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add scripts/coverage-check.sh for local coverage validation
- Configure pre-push hook in .pre-commit-config.yaml
- Update tasks.md with verification results from PR #3

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@montge
montge merged commit d505bf8 into master Dec 26, 2025
27 checks passed
montge added a commit that referenced this pull request Dec 26, 2025
Change complete and merged via PR #3.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants