Skip to content

Major Refactor: Caulking Python Rewrite#101

Closed
wz-gsa wants to merge 7 commits intomainfrom
python_refactor
Closed

Major Refactor: Caulking Python Rewrite#101
wz-gsa wants to merge 7 commits intomainfrom
python_refactor

Conversation

@wz-gsa
Copy link
Copy Markdown
Contributor

@wz-gsa wz-gsa commented Nov 7, 2025

Summary

This pull request modernizes Caulking from its original shell-based implementation into a maintainable, testable Python application that’s easier to extend, audit, and standardize across environments.

The goal of this rewrite is not to replace the spirit of the original tool — which served us well — but to carry its intent forward with stronger type safety, clearer modularity, and a cleaner developer experience.

Changes Proposed in This Pull Request

🧱 Core Rewrite

  • Converted Caulking’s logic from shell scripts to a proper Python 3 package using Typer for the CLI.

  • Introduced src/caulking/ with clean module boundaries:

    • installer.py, doctor.py, audit.py, config.py, manifest.py, and cli.py

  • Added pytest-based test coverage under tests/ with strict typing (mypy --strict) and security linting (bandit).

🧩 Modernized Hook Management

  • caulking smart-install detects repo languages (Python, Node, Go, Rust, etc.) and builds or merges a .pre-commit-config.yaml automatically.
  • Supports optional, language-specific add-on hooks (Ruff, Bandit, ESLint, TFLint, Hadolint, etc.).
  • Preserves existing configurations and merges new entries without overwriting.

🔐 Security & Compliance Enhancements

  • Gitleaks upgraded to baseline-free scanning, with a standardized ruleset in rules/gitleaks.toml.
  • Detect-secrets now runs in advisory mode to avoid false positives blocking commits.
  • Introduced a new caulking audit command to validate gitleaks installation, global hook enforcement, and overall secret-scanning health.

🩺 Diagnostics & Quality of Life

  • New caulking preflight command verifies that git, pre-commit, and gitleaks are properly installed and configured.

  • New caulking doctor module provides environment sanity checks and actionable hints.

  • Added Makefile with a consistent, developer-friendly command suite:

    • make lint, make type, make sec, make test, make audit, etc.

🧹 Legacy Compatibility

  • Legacy scripts are still available under scripts/ and can safely be uninstalled with:

    bash scripts/caulking-uninstall-legacy.sh --apply

  • The migration preserves configuration files and automatically backs up modified rc files.

🧪 Testing & Tooling

  • Introduces uv for reproducible environments and dependency pinning.
  • Adds dev extras for all quality gates: Ruff, Mypy, Bandit, Pytest, Coverage.
  • All subprocess calls are explicit and non-shell (shell=False), ensuring safe execution across platforms.

Repository Layout Before → After

Legacy Layout Modern Layout
Flat shell scripts under scripts/ Modular Python package under src/caulking/
Ad-hoc bash installers Typer-based CLI: caulking install, caulking audit, caulking smart-install
Manual gitleaks/detect-secrets setup Automated pre-commit scaffolding
No tests Pytest suite with coverage
Environment inconsistencies uv-based reproducible environments

Security Considerations

  • All subprocesses use fixed argument lists (shell=False), eliminating injection risks.
  • Bandit (MEDIUM/HIGH) and Ruff (security, bugbear, pycodestyle) rules enforced on every run.
  • Detect-secrets now runs advisory-only to prevent workflow blockers while maintaining awareness.
  • Gitleaks is mandatory and configured via a controlled ruleset.
  • Configuration edits are idempotent, reversible, and backed up with timestamps under ~/.caulking-smart-install-backup.

How to Test

uv sync --extra dev
make lint type sec test
python -m caulking preflight
python -m caulking smart-install --apply
python -m caulking audit

For legacy environments:

bash scripts/caulking-uninstall-legacy.sh --apply

Notes for Reviewers

  • This refactor preserves the same end goals as the original Caulking: protect secrets, enforce hygiene, and reduce friction.
  • The Python rewrite was built to extend that legacy safely — not to discard it — by making future maintenance simpler and compliance alignment easier.
  • Please review src/caulking/installer.py, src/caulking/cli.py, and the updated Makefile first to understand the new flow.

Acknowledgements

Huge thanks to the original Caulking authors — the design and intent of the original bash implementation directly informed this rewrite. This work builds on their foundation to ensure Caulking continues to be a reliable, lightweight defense mechanism for developers and security teams alike.

Would you like me to generate the corresponding commit message summary block (the single-line + extended format suitable for git commit --amend) to match this PR?

wz-gsa added 3 commits November 5, 2025 16:20
@wz-gsa
fix(hooks): quiet gitleaks and finalize pre-commit invocation
db9fe1a 
- Drop legacy --source and force `gitleaks git --pre-commit`.
- Add `--log-level=error` and `--no-color` to suppress noisy stderr.
- Keep `pass_filenames: false` to prevent filename injection.
@wz-gsa
refactor(core): modernize Caulking with Python-based enforcement, Ruf…
43e50ca 
…f-first precommit, and secure subprocess handling

This major refactor replaces legacy shell-based behavior with a maintainable, testable Python CLI. Key improvements:
- Introduces src/caulking/cli.py and config.py built with Typer.
- Adds SAFE_BINS validation, absolute-path enforcement, and shell=False subprocess calls for Bandit-safe execution.
- Implements global guarded pre-commit hook with Ruff-first enforcement and gitleaks fallback.
- Adds uninstall and bootstrap-client to transition cleanly from prior global hooks.
- Ensures compatibility with uv, pytest, Ruff, Bandit, and mypy (strict typing enabled).
- Updates pyproject.toml with types-PyYAML and dev extras for full lint/test coverage.
- Removes obsolete Concourse CI and shell scaffolding.

Validation:
- Ruff: clean
- mypy (strict): clean
- Bandit: clean with targeted nosec on validated calls
- pytest: all tests passing
@wz-gsa
feat: ecosystem-aware pre-commit planner + mandatory secret scans
46fff78
@wz-gsa wz-gsa self-assigned this Nov 7, 2025
@wz-gsa
ci: replace legacy workflows with modern CodeQL, CI, and pre-commit
d138d65 
Start fresh with a minimal, maintainable workflow set:

- Add .github/workflows/codeql.yml (Python-only analysis, minimal permissions).
- Add .github/workflows/ci.yml (ruff, mypy --strict, bandit, pytest via uv).
- Add .github/workflows/pre-commit.yml (run hooks on PRs).
- Drop outdated workflows tied to the prior bash-era layout.

Rationale:
- Old workflows didn’t cover CodeQL and no longer matched the repository shape.
- CodeQL was failing when .github/ disappeared and when Actions analysis misfired.
- Keeping CodeQL focused on Python avoids false failures and keeps runs fast.

Notes:
- Uses current non-deprecated actions: actions/checkout@v5, actions/setup-python@v6,
  actions/cache@v4, github/codeql-action@v4, astral-sh/setup-uv@v7.
- Permissions are minimal; no extra scopes granted.

Signed-off-by: William Zujkowski <william.zujkowski@gsa.gov>
@pburkholder
Copy link
Copy Markdown
Contributor

pburkholder commented Nov 7, 2025

THIS IS EXCITING! I'll try to set aside some time in the next week to test it out.

Instead of doing this as a PR, would it make sense to rename the existing repo to caulking-legacy and make a fresh start with a new repo?

@wz-gsa wz-gsa closed this Feb 12, 2026
@wz-gsa wz-gsa deleted the python_refactor branch February 13, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@wz-gsa wz-gsa

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wz-gsa @pburkholder