perf(server): non-running kernel packages scanned before filtering in application/json path

[knqyf263]

What did you do?

Sent a scan result with application/json to vuls server mode. The JSON contained an Ubuntu host with multiple installed kernel versions and RunningKernel.Release set to the running one.

What did you expect to happen?

Non-running kernel packages should be excluded before the heavy CVE detection (ospkg.Detect), as is the case with the text/plain server mode path and vuls scan (Remote/Local scan mode).

What happened instead?

From my reading of the code, it appears that filterCriterion in detector/vuls2/vendor.go#L278-L325 correctly filters non-running kernel packages by RunningKernel.Release. However, this filtering happens in postConvert — after ospkg.Detect has already performed the DB lookups and CVE matching for all packages.

On my environment, with 5 kernel versions (18 packages), application/json took ~5-9 seconds while the equivalent text/plain request took ~3.5 seconds. In a case I actually encountered, an Ubuntu 16.04 host had ~180 linux-aws kernel binary packages across dozens of versions (712 packages total), and the application/json scan took ~300 seconds, compared to ~5 seconds when the non-running kernel packages were removed from the input.

Analysis

As far as I can tell, there are currently three places where kernel packages are filtered:

1. scanner/debian.go:parseInstalledPackages() — Filters non-running kernel packages before scan. Called in Remote/Local scan mode (vuls scan) and text/plain server mode. Not called in application/json server mode path.

2. detector/vuls2/vendor.go:filterCriterion() — Filters non-running kernel packages after ospkg.Detect. Called in all paths. Ensures result correctness but does not reduce scan cost.

3. Previously in gost (gost/ubuntu.go#L124-L136, gost/debian.go#L93-L105) — Filtered non-running kernel source packages before each DB lookup in detectCVEsWithFixState(), skipping the lookup entirely for non-running kernels. These files were removed when detection moved to vuls2.

In the text/plain and Remote/Local scan paths, (1) removes non-running kernel packages before they reach ospkg.Detect, so there is no performance issue. In the application/json server mode path, the JSON is decoded directly into ScanResult and all packages are passed to ospkg.Detect without pre-filtering. The filtering only happens afterwards in (2).

Steps to reproduce the behaviour

1. Save the following as repro.json (5 kernel versions, only 4.4.0-1079-aws is running):

{
  "Family": "ubuntu",
  "Release": "16.04",
  "RunningKernel": {
    "Release": "4.4.0-1079-aws",
    "Version": "4.4.0-1079.89"
  },
  "Packages": {
    "linux-aws-headers-4.4.0-1070": { "Name": "linux-aws-headers-4.4.0-1070", "Version": "4.4.0-1070.80" },
    "linux-aws-headers-4.4.0-1075": { "Name": "linux-aws-headers-4.4.0-1075", "Version": "4.4.0-1075.85" },
    "linux-aws-headers-4.4.0-1079": { "Name": "linux-aws-headers-4.4.0-1079", "Version": "4.4.0-1079.89" },
    "linux-aws-headers-4.4.0-1083": { "Name": "linux-aws-headers-4.4.0-1083", "Version": "4.4.0-1083.93" },
    "linux-aws-headers-4.4.0-1090": { "Name": "linux-aws-headers-4.4.0-1090", "Version": "4.4.0-1090.101" },
    "linux-headers-4.4.0-1070-aws": { "Name": "linux-headers-4.4.0-1070-aws", "Version": "4.4.0-1070.80" },
    "linux-headers-4.4.0-1075-aws": { "Name": "linux-headers-4.4.0-1075-aws", "Version": "4.4.0-1075.85" },
    "linux-headers-4.4.0-1079-aws": { "Name": "linux-headers-4.4.0-1079-aws", "Version": "4.4.0-1079.89" },
    "linux-headers-4.4.0-1083-aws": { "Name": "linux-headers-4.4.0-1083-aws", "Version": "4.4.0-1083.93" },
    "linux-headers-4.4.0-1090-aws": { "Name": "linux-headers-4.4.0-1090-aws", "Version": "4.4.0-1090.101" },
    "linux-image-4.4.0-1070-aws": { "Name": "linux-image-4.4.0-1070-aws", "Version": "4.4.0-1070.80" },
    "linux-image-4.4.0-1075-aws": { "Name": "linux-image-4.4.0-1075-aws", "Version": "4.4.0-1075.85" },
    "linux-image-4.4.0-1079-aws": { "Name": "linux-image-4.4.0-1079-aws", "Version": "4.4.0-1079.89" },
    "linux-image-4.4.0-1083-aws": { "Name": "linux-image-4.4.0-1083-aws", "Version": "4.4.0-1083.93" },
    "linux-image-4.4.0-1090-aws": { "Name": "linux-image-4.4.0-1090-aws", "Version": "4.4.0-1090.101" },
    "linux-modules-4.4.0-1079-aws": { "Name": "linux-modules-4.4.0-1079-aws", "Version": "4.4.0-1079.89" },
    "linux-modules-4.4.0-1083-aws": { "Name": "linux-modules-4.4.0-1083-aws", "Version": "4.4.0-1083.93" },
    "linux-modules-4.4.0-1090-aws": { "Name": "linux-modules-4.4.0-1090-aws", "Version": "4.4.0-1090.101" }
  },
  "SrcPackages": {
    "linux-aws-4.4.0-1070.80": {
      "Name": "linux-aws", "Version": "4.4.0-1070.80",
      "BinaryNames": ["linux-aws-headers-4.4.0-1070", "linux-headers-4.4.0-1070-aws", "linux-image-4.4.0-1070-aws"]
    },
    "linux-aws-4.4.0-1075.85": {
      "Name": "linux-aws", "Version": "4.4.0-1075.85",
      "BinaryNames": ["linux-aws-headers-4.4.0-1075", "linux-headers-4.4.0-1075-aws", "linux-image-4.4.0-1075-aws"]
    },
    "linux-aws-4.4.0-1079.89": {
      "Name": "linux-aws", "Version": "4.4.0-1079.89",
      "BinaryNames": ["linux-aws-headers-4.4.0-1079", "linux-headers-4.4.0-1079-aws", "linux-image-4.4.0-1079-aws", "linux-modules-4.4.0-1079-aws"]
    },
    "linux-aws-4.4.0-1083.93": {
      "Name": "linux-aws", "Version": "4.4.0-1083.93",
      "BinaryNames": ["linux-aws-headers-4.4.0-1083", "linux-headers-4.4.0-1083-aws", "linux-image-4.4.0-1083-aws", "linux-modules-4.4.0-1083-aws"]
    },
    "linux-aws-4.4.0-1090.101": {
      "Name": "linux-aws", "Version": "4.4.0-1090.101",
      "BinaryNames": ["linux-aws-headers-4.4.0-1090", "linux-headers-4.4.0-1090-aws", "linux-image-4.4.0-1090-aws", "linux-modules-4.4.0-1090-aws"]
    }
  }
}

2. Compare scan times:

# Start vuls server
vuls server -listen localhost:5515

# application/json (all kernel packages sent to ospkg.Detect)
time curl -X POST -H "Content-Type: application/json" -d @repro.json http://localhost:5515/vuls > /dev/null

# text/plain (non-running kernel packages filtered before ospkg.Detect)
time curl -X POST \
  -H "Content-Type: text/plain" \
  -H "X-Vuls-OS-Family: ubuntu" \
  -H "X-Vuls-OS-Release: 16.04" \
  -H "X-Vuls-Kernel-Release: 4.4.0-1079-aws" \
  --data-binary "$(cat <<'EOF'
linux-aws-headers-4.4.0-1070,ii ,4.4.0-1070.80,linux-aws,4.4.0-1070.80
linux-aws-headers-4.4.0-1075,ii ,4.4.0-1075.85,linux-aws,4.4.0-1075.85
linux-aws-headers-4.4.0-1079,ii ,4.4.0-1079.89,linux-aws,4.4.0-1079.89
linux-aws-headers-4.4.0-1083,ii ,4.4.0-1083.93,linux-aws,4.4.0-1083.93
linux-aws-headers-4.4.0-1090,ii ,4.4.0-1090.101,linux-aws,4.4.0-1090.101
linux-headers-4.4.0-1070-aws,ii ,4.4.0-1070.80,linux-aws,4.4.0-1070.80
linux-headers-4.4.0-1075-aws,ii ,4.4.0-1075.85,linux-aws,4.4.0-1075.85
linux-headers-4.4.0-1079-aws,ii ,4.4.0-1079.89,linux-aws,4.4.0-1079.89
linux-headers-4.4.0-1083-aws,ii ,4.4.0-1083.93,linux-aws,4.4.0-1083.93
linux-headers-4.4.0-1090-aws,ii ,4.4.0-1090.101,linux-aws,4.4.0-1090.101
linux-image-4.4.0-1070-aws,ii ,4.4.0-1070.80,linux-aws,4.4.0-1070.80
linux-image-4.4.0-1075-aws,ii ,4.4.0-1075.85,linux-aws,4.4.0-1075.85
linux-image-4.4.0-1079-aws,ii ,4.4.0-1079.89,linux-aws,4.4.0-1079.89
linux-image-4.4.0-1083-aws,ii ,4.4.0-1083.93,linux-aws,4.4.0-1083.93
linux-image-4.4.0-1090-aws,ii ,4.4.0-1090.101,linux-aws,4.4.0-1090.101
linux-modules-4.4.0-1079-aws,ii ,4.4.0-1079.89,linux-aws,4.4.0-1079.89
linux-modules-4.4.0-1083-aws,ii ,4.4.0-1083.93,linux-aws,4.4.0-1083.93
linux-modules-4.4.0-1090-aws,ii ,4.4.0-1090.101,linux-aws,4.4.0-1090.101
EOF
)" http://localhost:5515/vuls > /dev/null

The detection results should be equivalent, but application/json is expected to be slower as ospkg.Detect processes all kernel packages before filterCriterion filters them out. The performance gap grows with the number of installed kernel versions.

Question for maintainers

I'd like to understand the project's intended approach for this case:

1. Apply kernel filtering before ospkg.Detect for the application/json server mode path — e.g. in preConvert() or in the server handler before calling DetectPkgCves(). This would make the application/json path consistent with text/plain in both behavior and performance. It would also be similar to how gost previously filtered non-running kernel packages before each DB lookup (gost/ubuntu.go#L124-L134).

2. Leave as-is and document that callers are responsible for filtering — The application/json server mode path is intended for use from external programs, and callers could be expected to send only running kernel packages. However, this creates a behavioral asymmetry where text/plain handles filtering internally but application/json does not, even though they are otherwise just different input formats for the same endpoint.

I'm leaning toward (1) since it aligns with the original gost design and keeps the two content types consistent, but I may be missing context on the design intent. I'd like to follow whichever approach best fits the project's direction.

Configuration

• Go version: go1.26.0 darwin/arm64
• Hash: c7e3461
• config.toml: default (no special configuration needed to reproduce)
• command: vuls server -listen localhost:5515

[knqyf263]

With the following files, I consistently see a 7-10x difference on my local environment and even more on production:

• repro-178pkgs.txt
• repro-178pkgs.json

# application/json
time curl -X POST -H "Content-Type: application/json" -d @repro-178pkgs.json http://localhost:5515/vuls > /dev/null

# text/plain
time curl -X POST \
  -H "Content-Type: text/plain" \
  -H "X-Vuls-OS-Family: ubuntu" \
  -H "X-Vuls-OS-Release: 16.04" \
  -H "X-Vuls-Kernel-Release: 4.4.0-1079-aws" \
  --data-binary @repro-178pkgs.txt \
  http://localhost:5515/vuls > /dev/null

[MaineK00n]

Thank you for the detailed analysis and reproduction steps.

The application/json endpoint is designed to receive JSON generated by vuls scan. In that workflow, non-running kernel packages are already filtered out during the scan phase by parseInstalledPackages()
(e.g., scanner/debian.go#L429-L484), so the performance issue you observed does not occur in the normal usage path.

The JSON in your reproduction was crafted externally and differs from what vuls produces in two ways:

1. Non-running kernel packages are included — vuls currently filters these out during the scan phase.
2. SrcPackages keys use - (e.g., "linux-aws-4.4.0-1070.80") — vuls uses the source package name as the key (e.g., "linux-aws") with BinaryNames aggregated under a single entry (see models/packages.go#L245, scanner/debian.go#L476). While filterCriterion currently happens to produce correct results with this non-standard key format, this behavior is not guaranteed and may break in future
   releases.

Regarding the suggestion to add kernel filtering in preConvert: we do not plan to do this. Ideally, vulnerabilities should be detected for all installed kernel packages regardless of whether they are
running, so that users can make informed decisions about removing unused vulnerable packages. The current filtering to the running kernel in parseInstalledPackages is a workaround for the SrcPackages key
collision issue (#1933), not the intended long-term behavior. Adding pre-filtering in preConvert would go against that direction.

If you are sending JSON from an external tool, please ensure it matches the format that vuls scan produces.

[knqyf263]

Thanks for your prompt answer. I understand the input JSON should be generated by vuls scan.