feat: add exploit-intelligence job polling, configuration & API endpoints

[Strum355]

Summary

Add the Exploit Intelligence orchestration module to trustify. This module provides API endpoints for the UI to trigger automated vulnerability reachability analysis via the EI client service (agent-morpheus-client), and a background worker that submits SBOMs, polls for results, and ingests returned CSAF VEX documents into trustify's advisory system.

Jira

Implements TC-4677

What changed

New module: exploit_intelligence (modules/fundamental/src/exploit_intelligence/)

• endpoints/ — HTTP handlers for triggering analysis and querying job status
• runner/ — background processing: SBOM upload with retry, polling for results, orphaned job recovery, heartbeat coordination
• service/ — CRUD operations on job and component entities, recovery queries
• model/ — request/response DTOs and API models
• auth.rs — OIDC client credentials and static token authentication for the EI service
• ei_client.rs — response type definitions for the EI client API

API endpoints:

• POST /v3/exploit-intelligence/analyze — trigger analysis for a CVE against an SBOM. Returns existing active job if one already exists (deduplication).
• GET /v3/exploit-intelligence/jobs — paginated job list, filterable by sbom_id and status
• GET /v3/exploit-intelligence/jobs/{id} — job details including per-component findings, advisory links, and report URLs

Analysis flows:

• CycloneDX (single-component): upload to EI → poll /api/v1/reports/by-scan-id/{scanId} → extract finding → ingest VEX if available
• SPDX (multi-component): upload to EI → poll /api/v1/products/{productId} → fetch per-component reports → extract findings and ingest VEX documents individually

Retry and polling:

• Upload retries use the backon crate (already a transitive dependency via walker-common) with configurable exponential backoff
• Polling uses backon::ConstantBuilder with a Cell-based consecutive failure counter for distinguishing transient errors from permanent failures
• All retry/poll parameters are configurable via environment variables using humantime format

Multi-instance safety:

• Heartbeat-based optimistic locking prevents multiple trustify instances from polling the same job. A heartbeat column (nanosecond timestamp) on the job table serves as a CAS lock — UPDATE ... WHERE heartbeat = <expected> ensures mutual exclusion.
• SELECT ... FOR UPDATE SKIP LOCKED during startup recovery atomically claims orphaned jobs
• CancellationToken for cooperative work cancellation when the heartbeat is lost
• All terminal status updates (update_job_completed, update_job_failed) use CAS against the heartbeat value — only the owning instance can transition job state
• Partial unique index (sbom_id, vulnerability_id) WHERE status IN ('pending', 'running') prevents duplicate active jobs at the database level

EI-generated advisory labeling:

• Ingested VEX documents are labeled source: exploit-intelligence to distinguish AI-generated findings from vendor-published advisories

Configuration (environment variables):

Variable	Description	Default
EXPLOIT_INTELLIGENCE_URL	Base URL of the EI client service	—
EXPLOIT_INTELLIGENCE_UI_URL	EI web UI base URL for report deep-links	—
EXPLOIT_INTELLIGENCE_POLL_INTERVAL	Polling interval (humantime)	30s
EXPLOIT_INTELLIGENCE_MAX_POLL_DURATION	Max polling duration (humantime)	30m
EXPLOIT_INTELLIGENCE_UPLOAD_MAX_RETRIES	Upload retry attempts	3
EXPLOIT_INTELLIGENCE_UPLOAD_RETRY_DELAY	Initial retry delay (humantime, exponential)	1s
EXPLOIT_INTELLIGENCE_MAX_CONSECUTIVE_POLL_FAILURES	Consecutive poll failures before abort	5
EXPLOIT_INTELLIGENCE_AUTH_TOKEN	Static auth token (fallback)	—
EXPLOIT_INTELLIGENCE_OIDC_TOKEN_URL	OIDC token endpoint	—
EXPLOIT_INTELLIGENCE_OIDC_CLIENT_ID	OIDC client ID	—
EXPLOIT_INTELLIGENCE_OIDC_CLIENT_SECRET	OIDC client secret	—

Migration changes (m0002230):

• Adds heartbeat DECIMAL NULL column to exploit_intelligence_job table
• Adds partial unique index idx_ei_job_active_dedup on (sbom_id, vulnerability_id) for active jobs

Dependencies

• Depends on TC-4675 / PR #2435 (entity and migration for job tracking)
• Adds backon as a direct workspace dependency (already transitive via walker-common)
• Adds humantime to the server crate for duration configuration
• Adds tokio-util to fundamental for CancellationToken

[sourcery-ai]

Sorry @Strum355, your pull request is larger than the review limit of 150000 diff characters

[ctron]

Please consider that this code can run multiple times on the same database, in multiple instances.

[ruromero]

The functionality is there and should work as expected but all the logic ended up in the endpoints module which is an anti-pattern. Look how other functionalities are implemented. I wonder why Claude didn't check for these patterns.

Another concerning point is the fact that there are many hardcoded values that could be configurable for operational reasons. I know there should be a balance between what should be configurable vs hardcoded. It's just something to take into account.

Pay attention at the long functions (+200 lines) doing multiple steps. This goes against the defined conventions.

Finally, the whole retry/polling logic can be moved to a common module so other parts of the code could benefit from it. I haven't checked if there are other candidates for future refactoring.

[ruromero]

Adding a distinction between RetryableFailure vs PermanentFailure would help end users distinguish between something that can be reattempted or something that requires an action before retry.

[Strum355]

That would require a lot of redesigning across the stack including the UI. Ultimately, all failures are retryable to some degree, Im not sure how much value add there is to adding a typed distinction that the error message itself doesnt show

[Strum355]

Please consider that this code can run multiple times on the same database, in multiple instances.

Thanks for pointing this out @ctron. Commit a7f1426 adds a heartbeat ownership mechanism similar to what exists for the ingestor. Theres a couple differences of course as the processes arent the same (EI jobs are request-driven), feel free to take a look and let me know if you come across any glaring issues with the implementation, or if anything could use inline clarification 🙂

[sourcery-ai]

Sorry @Strum355, your pull request is larger than the review limit of 150000 diff characters

[ctron]

Not sure we're using that pattern so far. Maybe check.

[Strum355]

What pattern specifically is this referring to? The endpoint path?

[ctron]

If EI is not available, how do we ensure those endpoints are not present?

[ctron]

Or, what's our strategy here?

[Strum355]

Is not mounting endpoints the approach we want to go with? Afaik that would be a first for the codebase, but Im fine with that if its an approach you feel strongly about. 503 Unavailable is an existing pattern in a few places in trustify, but the conditions are probably distinct enough that we can deviate here if you wish

[Strum355]

Ok, I just peeked a bit into the implementation. I think this is a bit overcomplicated. Why not follow a pattern like this: https://aminediro.com/posts/pg_job_queue/

Thanks for this, f272d7d should address this, similar to the importer I believe (but with the visibility timeout approach mentioned in that article instead of the heartbeat approach that importer uses). Some of the style comments were address in the next commit, so you might see some things that should be addressed that have been fixed later 😄 the EI job polling worker runs in the API profile which I hope is okay, outside the initial loading of the SBOM (to check for SBOM type to route the doc to the correct EI endpoint) this process should be fairly lightweight (majority of the time its just polling EI and then ingesting a very small VEX once the work is completed).