Improve first thought by RedTanny · Pull Request #291 · RHEcosystemAppEng/exploit-iq-agent

RedTanny · 2026-07-13T07:40:58Z

Summary

Refactors L1 intel gathering so the checker extracts CVE understanding and patch intel separately (per-file), then feeds focused context into Source Grep observations. Also includes related checker bugfixes and OSIDB identify improvements from this branch.

Why

The old single-prompt intel extraction mixed CVE narrative with full-patch analysis, which hurt keyword quality on large patches and gave observations little file-specific guidance. Splitting the flow improves search targeting and keeps large patches within the context budget.

High-level feature (intel refactor)

Prompt 1: CVE understanding (no patch required) — vulnerability type, component, ordered grep keywords
Prompt 2/3: Per-file patch analysis — functions/patterns per changed file, with token-budget batching for large patches
Result model: IntelGatheringResult aggregates per-file intel and converts back to VulnerabilityIntel for existing prompts/reports
Focused observations: when Source Grep targets a file, prepend that file’s intel into the observation context
New flow is now the sole intel path (dual-path flag removed)

Testing

Added unit coverage for the refactor and related empty/error tool handling:

test_file_change_summary.py — patch → per-file summaries / token estimates
test_cve_understanding_prompt.py — Prompt 1 formatting and keyword rules
test_patch_analysis_prompt.py — Prompt 2/3 formatting and functions summary helper
test_intel_gathering_result.py — lookup, aggregation, focused-context formatting
test_new_intel_flow.py — file priority + token-budget batch split
test_focused_observations.py — file filter extraction + focused context injection
test_check_empty_output.py — tool-error detection via status/prefix (not mid-content "error:")
tests/test_package_identifier.py — OSIDB affectedness / identify regression coverage

Bug fixes

Missing ADVISORY URL patterns / OSIDB in URL mining (b31b9d9): advisory/OSIDB URLs were missed when mining commit/advisory links for intel.
Weak hints continued into repo search (8ab9421): weak reference hints no longer keep searching repos; flow goes to the agent instead.
Noisy prompt regression (cafce28): removed extra prompt/intel fields that added noise and made the LLM less reliable.
“Patch file:” in report (47f8c01): dropped confusing “Patch file:” labeling from the checker report viewer.
Source Grep tool-call guidance (6c0576d): clarified grep tool instructions/schema so the LLM calls Source Grep correctly.
Identify regression (4132d2b): fixed PackageIdentifier affectedness regression after OSIDB support.

Also in this branch

OSIDB affectedness support in PackageIdentifier (55ac04d)
Stricter tool-error handling in observation/build agents so successful greps containing "error:" in source are not treated as tool failures

Test plan

Unit tests listed above
Local CVE run: Phoenix Initial_Intelligence_Gathering shows file_changes + CVE understanding
File-filtered Source Grep: observation span has focused_context_used=true
Spot-check known CVE verdicts for unexpected flips

… to agent

… and llm didn't follow prompt less accurate

Implements issues 01-06 of intel gathering refactor: - FileChangeSummary model and extraction (01) - file_changes span output (02) - CVE_UNDERSTANDING_PROMPT (03) - PATCH_ANALYSIS_PROMPT (04) - IntelGatheringResult object (05) - Dual-path intel gathering with enable_new_intel_flow config (06) Co-authored-by: Cursor <cursoragent@cursor.com>

…bservations

vbelouso · 2026-07-13T07:41:19Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Co-authored-by: Cursor <cursoragent@cursor.com>

zvigrinberg

@RedTanny Thank you for the PR.
Please see my comments.

zvigrinberg · 2026-07-13T08:31:11Z

    ReferenceHints,
    AdvisoryContent,
    GitSearchReport,
    CONFIDENCE_THRESHOLD_MIN,


@RedTanny This ( CONFIDENCE_THRESHOLD_MIN) is not being used at all.

zvigrinberg · 2026-07-13T08:37:38Z

+            prompt = PATCH_ANALYSIS_PROMPT.format(
+                vuln_id=vuln_id,
+                vulnerability_type=result.cve_understanding.vulnerability_type if result.cve_understanding else "",
+                affected_component=result.cve_understanding.affected_component if result.cve_understanding else "",
+                root_cause=result.cve_understanding.root_cause if result.cve_understanding else "",
+                functions_touched_summary=functions_summary,
+                patch_content=patch_content,
+            )


@RedTanny This is not being used at all..

What is being used is at

https://github.com/RHEcosystemAppEng/exploit-iq-agent/pull/291/changes#diff-7e2d863e8e84f6d82687b61d6ae8e02d8abb5b2592f130abcc7df14d362fad07R620-R627

So it's either a duplication with redundant var , or worse even a bug.

zvigrinberg · 2026-07-13T08:46:58Z

+
+        basename = Path(file_path).name.lower()
+        for fs in self.file_summaries:
+            if basename in fs.file_path.lower():


@RedTanny This is not safe enough, as for example both reasonable.c and sonable.c will match, while it's obvious these are two different files.
another example log.c prelog.c.
Maybe full check after transorming both sides to lowercases?

zvigrinberg · 2026-07-13T08:50:56Z

+            for fs in files:
+                # Per-file invocation to get structured output per file
+                file_prompt = PATCH_ANALYSIS_PROMPT.format(
+                    vuln_id=vuln_id,
+                    vulnerability_type=result.cve_understanding.vulnerability_type if result.cve_understanding else "",
+                    affected_component=result.cve_understanding.affected_component if result.cve_understanding else "",
+                    root_cause=result.cve_understanding.root_cause if result.cve_understanding else "",
+                    functions_touched_summary=format_functions_touched_summary([fs]),
+                    patch_content=_format_patch_content_for_files(parsed_patch, [fs]),
+                )
+                file_intel: PerFileIntel = await per_file_llm.ainvoke([SystemMessage(content=file_prompt)])
+                file_intel.file_path = fs.file_path  # Ensure path is set
+                results.append(file_intel)
+
+            span.set_output({
+                "files_analyzed": len(results),
+                "files": [r.file_path for r in results],
+            })
+            return results


@RedTanny Why running for each file LLM linearly ?
All files are independent, right?
So better IMO to create tasks for all files, and then run all tasks concurrently using asyncio.gather), it will improve performance and will make it much more rapid.

TamarW0 · 2026-07-20T10:36:26Z

/retest

exploit-iq-pac · 2026-07-20T10:36:30Z

Caution

There are some errors in your PipelineRun template.

PipelineRun	Error
vulnerability-analysis-on-pr	`CEL expression evaluation error: expression "event == \"pull_request\" &&\n!body.pull_request.draft &&\n(target_branch == \"main\" \|\| target_branch == \"rh-aiq-main\") &&\n(\"src/\".pathChanged() \|\| \"metrics_lib/\".pathChanged() \|\| \"pyproject.toml\".pathChanged() \|\| \"uv.lock\".pathChanged() \|\| \"Dockerfile\".pathChanged() \|\| \".dockerignore\".pathChanged())\n" failed to evaluate: no such key: pull_request`

TamarW0 · 2026-07-20T10:40:31Z

/test vulnerability-analysis-on-pr

RedTanny and others added 9 commits July 6, 2026 13:26

BugFix missing ADVISORY patterns and support for osidb in url minning

b31b9d9

BugFix when weak hints are found don't continue to repo search but go…

8ab9421

… to agent

Bugfix: reggression bug caused when adding to prompt cause more nosie…

cafce28

… and llm didn't follow prompt less accurate

remove Patch file: from report confuses the viewer

47f8c01

make LLM know how to call the grep tool correctly

6c0576d

feat: add OSIDB affectedness support to PackageIdentifier

55ac04d

fix regression issue identify

4132d2b

feat: switch intel gathering to per-file 3-prompt flow with focused o…

500b2aa

…bservations

RedTanny requested review from TamarW0 and zvigrinberg July 13, 2026 07:40

fix: align OSIDB NOTAFFECTED test with continue-to-verify status

2601f5a

Co-authored-by: Cursor <cursoragent@cursor.com>

zvigrinberg added enhancement New feature or request optimization labels Jul 13, 2026

zvigrinberg requested changes Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve first thought#291

Improve first thought#291
RedTanny wants to merge 10 commits into
RHEcosystemAppEng:mainfrom
RedTanny:improve-first-thought

RedTanny commented Jul 13, 2026

Uh oh!

vbelouso commented Jul 13, 2026 •

edited

Loading

Uh oh!

zvigrinberg left a comment

Uh oh!

zvigrinberg Jul 13, 2026 •

edited

Loading

Uh oh!

zvigrinberg Jul 13, 2026

Uh oh!

zvigrinberg Jul 13, 2026

Uh oh!

zvigrinberg Jul 13, 2026

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

exploit-iq-pac Bot commented Jul 20, 2026

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

RedTanny commented Jul 13, 2026

Summary

Why

High-level feature (intel refactor)

Testing

Bug fixes

Also in this branch

Test plan

Uh oh!

vbelouso commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

zvigrinberg left a comment

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

exploit-iq-pac Bot commented Jul 20, 2026

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vbelouso commented Jul 13, 2026 •

edited

Loading

zvigrinberg Jul 13, 2026 •

edited

Loading