Skip to content

feat(sca): 14 new lockfile parsers — pnpm, yarn, Pipfile, Gemfile, composer, NuGet, Dart, SwiftPM, Gradle, Maven, Mix, requirements.txt, pyproject.toml (closes #53)#130

Merged
Wolfvin merged 1 commit into
mainfrom
feat/issue-53-sca-lockfile-parsers
Jul 1, 2026
Merged

Conversation

@Wolfvin

@Wolfvin Wolfvin commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Issue #53 — SCA lockfile parsers (Phase 1)

Closes #53.

What this PR adds

14 new lockfile/manifest parsers under scripts/sca_parsers/, one module per format:

# File Format Ecosystem
1 pnpm_lock.py pnpm-lock.yaml (v6+v9) npm
2 yarn_lock.py yarn.lock (v1 / Berry / v2 PnP) npm
3 pipfile_lock.py Pipfile.lock pypi
4 gemfile_lock.py Gemfile.lock gem
5 composer_lock.py composer.lock (PHP) composer
6 packages_lock.py packages.lock.json (NuGet v1+v2) nuget
7 pubspec_lock.py pubspec.lock (Dart) pub
8 package_resolved.py Package.resolved (SwiftPM v1/v2/v3) swiftpm
9 gradle_lock.py gradle.lockfile + build.gradle gradle
10 pom_xml.py pom.xml (Maven) maven
11 mix_lock.py mix.lock (Elixir) hex
12 requirements_txt.py requirements.txt (pinned + unpinned + markers + URLs) pypi
13 pipfile.py Pipfile (declared deps) pypi
14 pyproject_toml.py pyproject.toml (PEP 621 + Poetry) pypi

Each parser exposes parse(path) -> List[Dependency] where Dependency is a dataclass with name / version / ecosystem / source_file / transitivity. Dispatch is via parse_lockfile(path) which auto-detects format by basename and returns (deps, ecosystem). Parsers fail gracefully — corrupt or unreadable files log a warning and return [] so the rest of the vuln-scan keeps working.

vulnscan_engine.py integration

  • DEPENDENCY_FILE_PATTERNS extended with 8 new ecosystems: gem, composer, nuget, pub, swiftpm, gradle, maven, hex. pnpm-lock.yaml also added to the npm lockfile list.
  • _parse_lock_file and _parse_manifest_file delegate to sca_parsers when the file basename is in the registry; otherwise they fall back to the existing inline parsers (no regression for npm / rust / pip / go).
  • Import is optional — if sca_parsers/ is removed, vuln-scan keeps working with the inline parsers.

Tests

  • 15 test fixtures under tests/fixtures/sca/ — one per format (Gemfile.lock + build.gradle + gradle.lockfile all included so both lockfile and manifest paths are exercised).
  • 59 new tests in tests/test_sca_parsers.py covering:
    • Per-parser unit tests (extraction, dedup, transitivity, ecosystem)
    • Dispatcher test (basename → ecosystem for all 15 formats)
    • Graceful-failure tests (corrupt JSON, empty file, unknown basename)
    • End-to-end scan_vulnerabilities() integration for the 3 DoD formats: Gemfile.lock, pnpm-lock.yaml, pubspec.lock

Definition of Done

  • scripts/sca_parsers/ with 14+ parser files
  • vuln-scan auto-detects Gemfile.lock, pnpm-lock.yaml, pubspec.lock
  • Test fixtures in tests/fixtures/sca/
  • Test suite green (1150 + 59 new = 1209 passing; 4 pre-existing failures unrelated to this PR — verified by stash+rerun on unmodified main)
  • All parsers reimplemented from public format specs (no code copied — LGPL/GPL compatibility preserved)
  • Graceful per-parser failure (try/except in dispatcher, no crash propagation)

Test results

tests/test_sca_parsers.py: 59 passed
tests/test_vuln_staleness.py: 35 passed, 4 skipped (no regression)
Full suite (excl. tests/test_integration.py — slow):
  1150 passed, 90 skipped, 4 pre-existing failures

The 4 pre-existing failures are unrelated to this PR:

  • test_codelensignore::test_actual_target_dir_is_ignored
  • test_universal_grammar_loader::test_auto_installs_when_env_var_set
  • test_universal_grammar_loader::test_returns_language_after_successful_install
  • test_universal_grammar_loader::test_available_languages_returns_list

All four fail identically on unmodified main (verified via git stash + rerun).

End-to-end sanity check

Copied all 15 SCA fixtures into a fresh temp workspace and ran scan_vulnerabilities(ws, offline=True):

status=ok
files_scanned=[Gemfile.lock, pnpm-lock.yaml, pubspec.lock, composer.lock,
  packages.lock.json, Package.resolved, gradle.lockfile, mix.lock,
  build.gradle, pom.xml, Pipfile, pyproject.toml, requirements.txt,
  yarn.lock, Pipfile.lock]
total_vulnerabilities=13  (npm + pip from existing VULN_DB entries)

New ecosystems (gem, maven, gradle, composer, nuget, pub, swiftpm, hex) have no VULN_DB entries yet — files are scanned without errors, ready for CVE entries to be added in a follow-up.

Notes

  • poetry.lock is intentionally not routed through sca_parsers — it is already handled by the existing inline _parse_poetry_lock() in vulnscan_engine.py and is not in the "14 new parsers" list from the issue.
  • pyproject_toml.py also exports parse_poetry_lock() for future consolidation if/when we decide to migrate poetry.lock to the modular package.

…mposer, NuGet, Dart, SwiftPM, Gradle, Maven, Mix, requirements.txt, pyproject.toml (closes #53)

Issue #53 — Phase 1 SCA lockfile parsers.

Adds scripts/sca_parsers/ package with one module per format:
  pnpm_lock, yarn_lock, pipfile_lock, pipfile, requirements_txt,
  pyproject_toml, gemfile_lock, composer_lock, packages_lock,
  pubspec_lock, package_resolved, gradle_lock, pom_xml, mix_lock.

Each parser exposes parse(path) -> List[Dependency] where Dependency is
a dataclass with name/version/ecosystem/source_file/transitivity.
Dispatch is via parse_lockfile(path) which auto-detects format by
basename and returns (deps, ecosystem). Parsers fail gracefully —
corrupt or unreadable files log a warning and return [] so the rest
of the vuln-scan keeps working.

vulnscan_engine.py wires the new parsers in:
  - DEPENDENCY_FILE_PATTERNS extended with 8 new ecosystems
    (gem, composer, nuget, pub, swiftpm, gradle, maven, hex) and
    pnpm-lock.yaml added to the npm lockfile list.
  - _parse_lock_file and _parse_manifest_file delegate to sca_parsers
    when the file basename is in the registry; otherwise they fall
    back to the existing inline parsers (no regression for npm/rust/
    pip/go).

15 test fixtures added under tests/fixtures/sca/. 59 new tests in
tests/test_sca_parsers.py cover every parser, the dispatcher, and
end-to-end scan_vulnerabilities() integration for the DoD formats
(Gemfile.lock, pnpm-lock.yaml, pubspec.lock).

Full test suite (excluding tests/test_integration.py which is slow):
  1150 passed, 90 skipped, 4 pre-existing failures (verified by
  stash+rerun that they fail identically on unmodified main).
  Combined with the 59 new tests, 1209 passing tests — well above the
  DoD threshold of 863.

All parsers reimplemented from public format specs (no code copied
from other projects — LGPL/GPL compatibility preserved).
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@sonarqubecloud

sonarqubecloud Bot commented Jul 1, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
3 Security Hotspots
D Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@Wolfvin Wolfvin merged commit f401bfc into main Jul 1, 2026
3 of 10 checks passed
@Wolfvin Wolfvin deleted the feat/issue-53-sca-lockfile-parsers branch July 1, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] SCA lockfile parser expansion + dependency-aware rules (14 new parsers, project-depends-on)

1 participant