feat(sca): 14 new lockfile parsers — pnpm, yarn, Pipfile, Gemfile, composer, NuGet, Dart, SwiftPM, Gradle, Maven, Mix, requirements.txt, pyproject.toml (closes #53)#130
Merged
Conversation
…mposer, NuGet, Dart, SwiftPM, Gradle, Maven, Mix, requirements.txt, pyproject.toml (closes #53) Issue #53 — Phase 1 SCA lockfile parsers. Adds scripts/sca_parsers/ package with one module per format: pnpm_lock, yarn_lock, pipfile_lock, pipfile, requirements_txt, pyproject_toml, gemfile_lock, composer_lock, packages_lock, pubspec_lock, package_resolved, gradle_lock, pom_xml, mix_lock. Each parser exposes parse(path) -> List[Dependency] where Dependency is a dataclass with name/version/ecosystem/source_file/transitivity. Dispatch is via parse_lockfile(path) which auto-detects format by basename and returns (deps, ecosystem). Parsers fail gracefully — corrupt or unreadable files log a warning and return [] so the rest of the vuln-scan keeps working. vulnscan_engine.py wires the new parsers in: - DEPENDENCY_FILE_PATTERNS extended with 8 new ecosystems (gem, composer, nuget, pub, swiftpm, gradle, maven, hex) and pnpm-lock.yaml added to the npm lockfile list. - _parse_lock_file and _parse_manifest_file delegate to sca_parsers when the file basename is in the registry; otherwise they fall back to the existing inline parsers (no regression for npm/rust/ pip/go). 15 test fixtures added under tests/fixtures/sca/. 59 new tests in tests/test_sca_parsers.py cover every parser, the dispatcher, and end-to-end scan_vulnerabilities() integration for the DoD formats (Gemfile.lock, pnpm-lock.yaml, pubspec.lock). Full test suite (excluding tests/test_integration.py which is slow): 1150 passed, 90 skipped, 4 pre-existing failures (verified by stash+rerun that they fail identically on unmodified main). Combined with the 59 new tests, 1209 passing tests — well above the DoD threshold of 863. All parsers reimplemented from public format specs (no code copied from other projects — LGPL/GPL compatibility preserved).
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.




Issue #53 — SCA lockfile parsers (Phase 1)
Closes #53.
What this PR adds
14 new lockfile/manifest parsers under
scripts/sca_parsers/, one module per format:pnpm_lock.pypnpm-lock.yaml(v6+v9)yarn_lock.pyyarn.lock(v1 / Berry / v2 PnP)pipfile_lock.pyPipfile.lockgemfile_lock.pyGemfile.lockcomposer_lock.pycomposer.lock(PHP)packages_lock.pypackages.lock.json(NuGet v1+v2)pubspec_lock.pypubspec.lock(Dart)package_resolved.pyPackage.resolved(SwiftPM v1/v2/v3)gradle_lock.pygradle.lockfile+build.gradlepom_xml.pypom.xml(Maven)mix_lock.pymix.lock(Elixir)requirements_txt.pyrequirements.txt(pinned + unpinned + markers + URLs)pipfile.pyPipfile(declared deps)pyproject_toml.pypyproject.toml(PEP 621 + Poetry)Each parser exposes
parse(path) -> List[Dependency]whereDependencyis a dataclass withname / version / ecosystem / source_file / transitivity. Dispatch is viaparse_lockfile(path)which auto-detects format by basename and returns(deps, ecosystem). Parsers fail gracefully — corrupt or unreadable files log a warning and return[]so the rest of the vuln-scan keeps working.vulnscan_engine.py integration
DEPENDENCY_FILE_PATTERNSextended with 8 new ecosystems:gem,composer,nuget,pub,swiftpm,gradle,maven,hex.pnpm-lock.yamlalso added to the npm lockfile list._parse_lock_fileand_parse_manifest_filedelegate tosca_parserswhen the file basename is in the registry; otherwise they fall back to the existing inline parsers (no regression for npm / rust / pip / go).sca_parsers/is removed, vuln-scan keeps working with the inline parsers.Tests
tests/fixtures/sca/— one per format (Gemfile.lock + build.gradle + gradle.lockfile all included so both lockfile and manifest paths are exercised).tests/test_sca_parsers.pycovering:scan_vulnerabilities()integration for the 3 DoD formats:Gemfile.lock,pnpm-lock.yaml,pubspec.lockDefinition of Done
scripts/sca_parsers/with 14+ parser filesvuln-scanauto-detectsGemfile.lock,pnpm-lock.yaml,pubspec.locktests/fixtures/sca/Test results
The 4 pre-existing failures are unrelated to this PR:
test_codelensignore::test_actual_target_dir_is_ignoredtest_universal_grammar_loader::test_auto_installs_when_env_var_settest_universal_grammar_loader::test_returns_language_after_successful_installtest_universal_grammar_loader::test_available_languages_returns_listAll four fail identically on unmodified
main(verified viagit stash+ rerun).End-to-end sanity check
Copied all 15 SCA fixtures into a fresh temp workspace and ran
scan_vulnerabilities(ws, offline=True):New ecosystems (gem, maven, gradle, composer, nuget, pub, swiftpm, hex) have no VULN_DB entries yet — files are scanned without errors, ready for CVE entries to be added in a follow-up.
Notes
poetry.lockis intentionally not routed throughsca_parsers— it is already handled by the existing inline_parse_poetry_lock()invulnscan_engine.pyand is not in the "14 new parsers" list from the issue.pyproject_toml.pyalso exportsparse_poetry_lock()for future consolidation if/when we decide to migratepoetry.lockto the modular package.