feat: implement cpe matching and nvd importer

[rlang-ta]

Motivation:
Today an SBOM mostly matches advisories through PURLS. For embedded components or small ecosystems, matching by CPE remains the only good tool for now. Hence this PR aims to extend trustify with matching by CPE as well as an importer for ingesting from NVD.

Changes

• cpe_status table (source-neutral: vendor/product identity + affected version_range), keyed to a vulnerability
• CVE loader now ingests CNA + ADP affected[].cpes; SPDX cpe23Type external refs ingested
• Matching in SBOM details + batch severity counts, and the reverse backlink so a vulnerability page lists SBOMs it matches via a package CPE (not just via PURL)

NVD importer

• New Nvd importer kind pulling per-year release assets (CVE-.json.xz + .meta) from the fkie-cad/nvd-json-data-feeds mirror; incremental re-ingest via the .meta sha256 so steady-state runs only transfer changed years
• NvdLoader maps configurations → cpeMatch bounds into cpe_status + version_range
• Key detail: NVD ranges are stored under VersionScheme::Semver so version_matches does ordered range comparison, not the exact-only matching the generic scheme uses
• Format::NVD is only used when passed explicitly (a bare NVD cve object is indistinguishable from OSV by content sniffing), wired through the DocumentDetector

Schema

• New migration m0002240_create_cpe_status (reversible; FK cascade on advisory/vulnerability)

Summary by Sourcery

Implement CPE-based vulnerability applicability across ingestion and matching, including new CPE parsing and storage primitives, and add an NVD importer that feeds semver-range CPE status into SBOM and vulnerability views.

New Features:

• Add CPE-based vulnerability matching for SBOMs using package-level CPE identities and version ranges alongside existing PURL and product-status matching
• Introduce an NVD importer and loader that ingest per-year NVD CVE feeds, map configurations to semver-based CPE status ranges, and expose NVD as an advisory format
• Support parsing CPE 2.3 strings, normalizing CPE versions to ANY for identity keys, and ingesting SPDX cpe23Type references into SBOM package data

Bug Fixes:

• Align SBOM severity batch counts with SBOM details by including CPE-based matches so vulnerabilities found via CPE are consistently reflected

Enhancements:

• Extend vulnerability detail views to backlink SBOMs that match via CPE-based applicability, not just PURL or product-status matches
• Augment CVE ingestion to harvest CNA and ADP CPE/version data into cpe_status, ensuring richer applicability for embedded components

Tests:

• Add end-to-end tests covering CPE-based SBOM matching, severity aggregation parity, NVD-derived semver range matching, SPDX CPE ingestion, and vulnerability-to-SBOM backlinks via CPE

Chores:

• Add a new cpe_status database table and corresponding migration and entity model to store CPE-keyed advisory status with version ranges

[sourcery-ai]

Reviewer's Guide

Implements CPE-based vulnerability matching end-to-end (CVE and NVD ingestion, schema, SQL, and backlinks) and adds an NVD importer that pulls per-year JSON feeds, mapping NVD configurations into semver-backed CPE status ranges used in SBOM and vulnerability views.

Entity relationship diagram for new cpe_status-based CPE matching

erDiagram
  advisory {
    uuid id
  }

  vulnerability {
    text id
  }

  status {
    uuid id
    text slug
  }

  cpe {
    uuid id
    text vendor
    text product
    text part
    text version
  }

  version_range {
    uuid id
    text low_version
    text high_version
    uuid version_scheme_id
  }

  cpe_status {
    uuid id
    uuid advisory_id
    text vulnerability_id
    uuid status_id
    uuid cpe_id
    uuid version_range_id
    uuid context_cpe_id
  }

  advisory ||--o{ cpe_status : has
  vulnerability ||--o{ cpe_status : has
  status ||--o{ cpe_status : classifies
  cpe ||--o{ cpe_status : identifies
  version_range ||--o{ cpe_status : bounds

Loading

File-Level Changes

Change	Details	Files
Extend CVE loader to ingest CPE applicability and store it as cpe_status rows keyed by vendor/product identity plus version ranges, shared with PURL logic.	Introduce CpeStatusCreator and CpeStatusEntry to batch-insert CPE-based status rows with deterministic UUIDs and shared VersionInfo handling. Parse affected[].cpes from CNA and ADP containers, normalize CPEs to identity (any-version), derive VersionSpec and status via new helpers, and fall back to CPE’s own version when no explicit versions list is present. Batch-create referenced CPE identities via CpeCreator alongside base PURLs and ensure idempotent status creation with tests covering CNA-only, ADP-only, and re-ingest scenarios. Refactor CVE version/status mapping into version_spec_and_status and status_slug helpers used by both PURL and CPE paths.	modules/ingestor/src/service/advisory/cve/loader.rs modules/ingestor/src/graph/cpe_status_creator.rs modules/ingestor/src/graph/advisory/cpe_status.rs entity/src/cpe_status.rs migration/src/m0002240_create_cpe_status.rs
Enhance CPE handling to support CPE 2.3 strings and to derive vendor/product identity CPEs by normalizing versions to ANY.	Implement Cpe::with_any_version to reconstruct a version-ANY URI while preserving vendor/product and key components, dropping trailing ANY segments and handling packed edition attributes. Add CPE 2.3 parsing by converting cpe:2.3 strings into CPE 2.2 URIs, including escape handling, wildcard mapping, and packing extended attributes into edition. Introduce helper functions for splitting/unescaping CPE 2.3 components and percent-encoding them into URI syntax, with extensive tests validating UUID stability, wildcard handling, extended attributes, language, and error cases. Extend SPDX ingestor to accept both cpe22Type and cpe23Type external references and store them as CPE references on SBOM packages.	common/src/cpe.rs modules/ingestor/src/graph/sbom/spdx.rs
Wire CPE-based matching into SBOM details and batch severity counts using cpe_status and package-level CPEs harvested from SBOMs.	Add raw_sql::cpe_advisory_info_sql to join sbom_node_cpe_ref with cpe_status via vendor/product application CPE identity and version_matches on package version, returning rows compatible with product_advisory_info_sql. Update SbomDetails.fetch_sbom_details to execute both product_advisory_info_sql and cpe_advisory_info_sql and merge their IdSets so advisories found only via CPE are included in SBOM detail views. Extend batch_severity_counts_sql with sbom_cpe_pkgs and cpe_version_matches CTEs, unioning CPE identity matches into all_affected so SBOM list severity counts reflect CPE-matched vulnerabilities. Add end-to-end SBOM tests for CPE-based matching, severity count parity, NVD CPE range matching, and SPDX ingestion of cpe23 firmware SBOM, including positive and negative match scenarios and handling of junk CPEs.	modules/fundamental/src/sbom/model/raw_sql.rs modules/fundamental/src/sbom/model/details.rs modules/fundamental/tests/sbom/details.rs modules/fundamental/tests/sbom/spdx.rs
Add reverse backlinks from vulnerabilities to SBOMs matched via CPE and ensure NVD-derived CPE applicability participates in vulnerability views.	Extend VulnerabilityAdvisorySummary building to run a CPE-based raw SQL query over cpe_status, joining advisory, status, sbom_node_cpe_ref, sbom_package, sbom_node_purl_ref, and describing SBOM nodes, and add the resulting SbomStatusCatcher entries. Add tests verifying that an NVD advisory whose CPE range matches a firmware SBOM via BusyBox CPE is visible in the vulnerability’s related SBOM list and guards against the earlier asymmetry where SBOM pages showed the vuln but vuln pages did not backlink. Ensure status filtering excludes not_affected and that version_matches uses package CPE version or package version as fallback when CPE version is wildcard.	modules/fundamental/src/vulnerability/model/details/vulnerability_advisory.rs modules/fundamental/tests/vuln/mod.rs
Introduce NVD ingestion pipeline: importer configuration, per-year feed runner, NVD format detection, schema, loader, and tests mapping NVD configurations into semver CPE ranges.	Define NvdImporter config with source URL, explicit years, and optional startYear; document default GitHub mirror and integrate into ImporterConfiguration enum and deref implementations. Add ImportRunner::run_once_nvd and associated runner module that resolves years to process, fetches per-year .meta and .json.xz assets, performs sha256-based incremental re-ingest, decompresses feeds, and invokes IngestorService.ingest with Format::NVD for each CVE fragment. Register Format::NVD in ingestor service, extend DocumentDetector to parse NvdCve when Format::NVD is explicitly provided, and wire NvdLoader into ingestion dispatch. Implement NvdCve and NvdYearFeed minimal serde schemas that use serde_json::RawValue for per-CVE fragments and map metrics, weaknesses, configurations, and cpeMatch fields needed by the loader. Implement NvdLoader that upserts vulnerabilities (without touching authoritative_advisory_id), creates NVD advisory records with issuer NVD, parses CVSS v2/v3/v4 vectors, chooses best base score, and maps NVD configurations.cpeMatch into cpe_status entries with VersionScheme::Semver and appropriate VersionSpec ranges or exacts, plus tests validating semver range storage and idempotent re-ingest. Add NVD importer runner wiring, progress reporting, and meta sha parsing, plus resolve_years helper that uses explicit years or startYear..current-year with NVD_FIRST_YEAR default. Update openapi and importer JSON schema to expose NVD importer type, format option, and configuration properties.	modules/importer/src/model/nvd.rs modules/importer/src/model/mod.rs modules/importer/src/runner/nvd/mod.rs modules/importer/src/runner/mod.rs modules/ingestor/src/service/format.rs modules/ingestor/src/service/detect.rs modules/ingestor/src/service/advisory/nvd/loader.rs modules/ingestor/src/service/advisory/nvd/schema.rs modules/ingestor/src/service/advisory/mod.rs modules/importer/schema/importer.json xtask/schema/generate-dump.json openapi.yaml etc/test-data/nvd/CVE-2099-1000.json etc/test-data/nvd/CVE-2099-2000.json etc/test-data/nvd/CVE-2099-2001.json
Add cpe_status persistence and migration, and enable schema-level support for CPE status queries.	Create cpe_status entity model with relations to version_range, cpe, vulnerability, advisory, status, and advisory_vulnerability, including optional context_cpe. Add migration m0002240_create_cpe_status that creates the cpe_status table with appropriate foreign keys, cascade behavior on advisory deletion, and indexes on cpe_id and (advisory_id, vulnerability_id). Register the new migration in the global Migrator and expose cpe_status module from entity crate. Ensure CpeStatusCreator uses version_range entity for range persistence and deduplicates both ranges and status rows via deterministic v5 UUIDs.	entity/src/cpe_status.rs migration/src/m0002240_create_cpe_status.rs migration/src/lib.rs
Minor supporting changes needed for the new functionality (serde_json feature, importer wiring, and SPDX tests).	Enable serde_json "raw_value" feature in Cargo.toml so NVD per-CVE fragments can be held as RawValue in feed schema. Expose new modules (nvd runner, nvd service, cpe_status graph module) in respective mod.rs files and hook NVD and CPE components into existing graphs and services. Add synthetic test data files for CVE v5 fixtures, NVD fixtures, and an SPDX firmware SBOM with CPE23 external references used across new tests.	Cargo.toml modules/importer/src/runner/mod.rs modules/importer/src/model/mod.rs modules/ingestor/src/graph/mod.rs modules/ingestor/src/graph/advisory/mod.rs modules/ingestor/src/service/advisory/mod.rs modules/ingestor/src/graph/sbom/spdx.rs etc/test-data/cve/CVE-2099-0001.json etc/test-data/cve/CVE-2099-0002.json etc/test-data/cve/CVE-2099-0003.json etc/test-data/spdx/cpe23-firmware.json

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 2 issues, and left some high level feedback:

• The CPE normalization helper Cpe::with_any_version silently falls back to returning the original CPE if reconstruction fails; consider returning a Result or at least emitting a log so unexpected parse failures don’t get hidden and are easier to diagnose.
• The new CPE-based matching SQL (cpe_advisory_info_sql and the CTEs added to batch_severity_counts_sql) duplicates several concepts and joins; factoring common pieces (e.g. sbom_cpe_pkgs, CPE identity joins) into a shared helper or view would reduce the risk of the two paths drifting out of sync.
• In the NVD importer, resolve_years gives precedence to years over start_year but this interaction isn’t obvious from the type or config shape; consider validating and/or explicitly documenting precedence so misconfigurations (e.g. both set with conflicting values) are easier to understand.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The CPE normalization helper `Cpe::with_any_version` silently falls back to returning the original CPE if reconstruction fails; consider returning a `Result` or at least emitting a log so unexpected parse failures don’t get hidden and are easier to diagnose.
- The new CPE-based matching SQL (`cpe_advisory_info_sql` and the CTEs added to `batch_severity_counts_sql`) duplicates several concepts and joins; factoring common pieces (e.g. `sbom_cpe_pkgs`, CPE identity joins) into a shared helper or view would reduce the risk of the two paths drifting out of sync.
- In the NVD importer, `resolve_years` gives precedence to `years` over `start_year` but this interaction isn’t obvious from the type or config shape; consider validating and/or explicitly documenting precedence so misconfigurations (e.g. both set with conflicting values) are easier to understand.

## Individual Comments

### Comment 1
<location path="modules/fundamental/tests/sbom/details.rs" line_range="379-213" />
<code_context>
+async fn sbom_details_nvd_cpe_range_matching(ctx: &TrustifyContext) -> Result<(), anyhow::Error> {
</code_context>
<issue_to_address>
**suggestion (testing):** Consider additional NVD range edge cases: unbounded ranges and exact boundary matches

The current test covers versions strictly inside and outside a bounded range, but misses two important NVD range patterns:

- Unbounded ranges with only a start or end bound (e.g. `>= 1.0.0` or `< 2.0.0`). A fixture using `versionStartIncluding` or `versionEnd*` without the other bound would validate `Version::Unbounded` handling in `version_spec`/`version_matches`.
- Exact boundary cases where the package version equals an inclusive or exclusive bound. Assertions for versions equal to `versionStartIncluding` and `versionEndExcluding` would help catch off‑by‑one errors.

Adding targeted fixtures and assertions for these cases would significantly strengthen the semver-range tests.

Suggested implementation:

```rust
#[test_context(TrustifyContext)]
#[test(tokio::test)]
#[instrument]
async fn sbom_details_nvd_cpe_range_matching(ctx: &TrustifyContext) -> Result<(), anyhow::Error> {
    let sbom = SbomService::new(PaginationCache::for_test());

    let result = ctx.ingest_document("spdx/cpe23-firmware.json").await?;

    // Bounded range: version inside vs outside the range
    let hit = ctx
        .ingest_document_as("nvd/CVE-2099-2000.json", Format::NVD, ("source", "nvd"))
        .await?;
    assert_eq!(hit.document_id, Some("CVE-2099-2000".to_string()));

    let miss = ctx
        .ingest_document_as("nvd/CVE-2099-2001.json", Format::NVD, ("source", "nvd"))
        .await?;
    assert_eq!(miss.document_id, Some("CVE-2099-2001".to_string()));

    // Unbounded range: only start bound (e.g. versionStartIncluding, no versionEnd*)
    // This fixture should contain a CPE range like "versionStartIncluding": "1.0.0"
    // and match a component version >= 1.0.0.
    let unbounded_start_hit = ctx
        .ingest_document_as(
            "nvd/CVE-2099-2002-unbounded-start.json",
            Format::NVD,
            ("source", "nvd"),
        )
        .await?;
    assert_eq!(
        unbounded_start_hit.document_id,
        Some("CVE-2099-2002".to_string())
    );

    // Unbounded range: only end bound (e.g. versionEndExcluding, no versionStart*)
    // This fixture should contain a CPE range like "versionEndExcluding": "2.0.0"
    // and match a component version < 2.0.0.
    let unbounded_end_hit = ctx
        .ingest_document_as(
            "nvd/CVE-2099-2003-unbounded-end.json",
            Format::NVD,
            ("source", "nvd"),
        )
        .await?;
    assert_eq!(
        unbounded_end_hit.document_id,
        Some("CVE-2099-2003".to_string())
    );

    // Exact boundary (inclusive): package version == versionStartIncluding
    // This fixture should contain a CPE range like "versionStartIncluding": "1.0.0"
    // with the SBOM component at exactly 1.0.0, and must be treated as a hit.
    let inclusive_boundary_hit = ctx
        .ingest_document_as(
            "nvd/CVE-2099-2004-inclusive-start.json",
            Format::NVD,
            ("source", "nvd"),
        )
        .await?;
    assert_eq!(
        inclusive_boundary_hit.document_id,
        Some("CVE-2099-2004".to_string())
    );

    // Exact boundary (exclusive): package version == versionEndExcluding
    // This fixture should contain a CPE range like "versionEndExcluding": "2.0.0"
    // with the SBOM component at exactly 2.0.0, and must be treated as a miss.
    let exclusive_boundary_miss = ctx
        .ingest_document_as(
            "nvd/CVE-2099-2005-exclusive-end.json",
            Format::NVD,
            ("source", "nvd"),
        )
        .await?;
    assert_eq!(
        exclusive_boundary_miss.document_id,
        Some("CVE-2099-2005".to_string())
    );

```

1. Add the corresponding NVD JSON fixtures referenced above (`nvd/CVE-2099-2002-unbounded-start.json`, `nvd/CVE-2099-2003-unbounded-end.json`, `nvd/CVE-2099-2004-inclusive-start.json`, `nvd/CVE-2099-2005-exclusive-end.json`) under your test data directory. Each should define appropriate `cpeMatch` entries:
   - Unbounded start: `versionStartIncluding` set, no `versionEnd*`.
   - Unbounded end: `versionEndExcluding` (or `versionEndIncluding`) set, no `versionStart*`.
   - Inclusive boundary: `versionStartIncluding` equal to the SBOM component version.
   - Exclusive boundary: `versionEndExcluding` equal to the SBOM component version.
2. In the part of `sbom_details_nvd_cpe_range_matching` where you currently assert matches/misses for `hit` and `miss` (e.g. via `sbom.details(..)` or equivalent), add parallel assertions for the new variables:
   - `unbounded_start_hit` and `unbounded_end_hit` should produce vulnerabilities for the relevant SBOM component (i.e. `version_matches` must return true).
   - `inclusive_boundary_hit` should be treated as a hit at the lower bound.
   - `exclusive_boundary_miss` should *not* produce a match at the upper exclusive bound.
3. Ensure these new assertions exercise the same `version_spec` / `version_matches` code path as the existing test, so that off-by-one and unbounded range handling is covered by this test.
</issue_to_address>

### Comment 2
<location path="modules/fundamental/tests/sbom/spdx.rs" line_range="171-180" />
<code_context>
+async fn ingest_spdx_cpe23_refs(ctx: &TrustifyContext) -> Result<(), anyhow::Error> {
</code_context>
<issue_to_address>
**suggestion (testing):** Add a test that exercises SPDX `cpe23Type` with extended attributes and language

The existing `ingest_spdx_cpe23_refs` test only covers simple application CPEs. Since the parser and SPDX ingestion now support extended attributes and language, please add a case in `spdx/cpe23-firmware.json` where `cpe23Type` includes a non-empty language (e.g. `en-US`) and a meaningful extended attribute (such as `target_hw` or `sw_edition`). Then assert that `fetch_sbom_packages` returns the expected normalized CPE URI, including packed edition and correct language normalization, to validate the full end-to-end handling of richer 2.3 CPEs.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[rh-jfuller]

A few initial observations to get started:

• maybe we consider (in the future) adding a composite index on cpe(part, vendor, product), as CPE table grows (NVD feeds will add thousands of CPE identity rows) joins will prob degrade to seq scans.
• It looks like unparseable CPEs in NVD loader are silently dropped - maybe add some logging ?

thx for the great submission (I appreciate how you have broken up commits usefully) ... more observations to come.

[codecov]

Codecov Report

❌ Patch coverage is 84.89461% with 129 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.17%. Comparing base (9d0cd82) to head (1d25648).

Files with missing lines	Patch %	Lines
modules/ingestor/src/graph/advisory/cpe_status.rs	23.68%	28 Missing and 1 partial ⚠️
...odules/ingestor/src/service/advisory/nvd/loader.rs	75.43%	22 Missing and 6 partials ⚠️
common/src/cpe.rs	91.03%	7 Missing and 13 partials ⚠️
entity/src/cpe_status.rs	0.00%	18 Missing ⚠️
modules/importer/src/runner/nvd/mod.rs	89.91%	2 Missing and 10 partials ⚠️
modules/importer/src/model/nvd.rs	0.00%	9 Missing ⚠️
entity/src/cpe.rs	91.39%	7 Missing and 1 partial ⚠️
modules/ingestor/src/graph/cpe_status_creator.rs	91.66%	2 Missing ⚠️
...odules/ingestor/src/service/advisory/cve/loader.rs	98.36%	1 Missing ⚠️
modules/ingestor/src/service/detect.rs	50.00%	0 Missing and 1 partial ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2518      +/-   ##
==========================================
+ Coverage   71.67%   72.17%   +0.50%     
==========================================
  Files         457      465       +8     
  Lines       27743    28589     +846     
  Branches    27743    28589     +846     
==========================================
+ Hits        19884    20635     +751     
- Misses       6713     6772      +59     
- Partials     1146     1182      +36     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jcrossley3]

It's a pretty big PR, but I appreciate the commit organization. I can't really comment on the raw sql, but the importer looks reasonable. Maybe add an NVD entry to the sample data with reasonable defaults to enable someone to easily try it?

[rlang-ta]

Hey @jcrossley3,

thanks for having a look at it. I'm aware that the PR exploded in size and I'm sorry about that. I couldn't find a proper way to split the PR in reasonable chunks without burdening you folks with a lot of PRs and context switching :/ (except maybe making the nvd importer a self contained PR)

Anyway i followed up on you suggestion and added

• the composite index. I did not do any performance checks to be honest, but i have no baseline to compare against. Let me know though if thats a concern for you
• A warning now gets logged for dropped NVD entries
• A warning gets logged in case an NVD entry has both years and start_year as stated by sourcery
• Sample data starting from 2024, matching the rest of the samples

[rh-jfuller]

/scale-test

[github-actions]

🛠️ Scale test has started! Follow the progress here: Workflow Run