feat: integrate LangChain agent into prejudge pipeline#13
Conversation
Signed-off-by: LeiGuo622 <2931505649@qq.com>
There was a problem hiding this comment.
Pull request overview
This PR integrates an LLM (LangChain) “judge agent” into the prejudge pipeline as the final decision step for determining whether a patch needs backporting by checking if vulnerable code exists in the downstream target.
Changes:
- Added an LLM-based final check (
judge_agent_llm) toPrejudgeController.analyze_and_report. - Introduced a new
JudgeAgentLangChain executor with tools/prompting and decision parsing. - Added
prejudgepackage initializer exportingJudgeAgentandPrejudgeController.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
src/prejudge/prejudge.py |
Adds a new LLM-agent judgment step to produce the final true/false output. |
src/prejudge/judge_agent.py |
Implements the LangChain-based agent, model configuration, patch retrieval, and response parsing. |
src/prejudge/__init__.py |
Declares prejudge as a package and re-exports key entry points. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from judge_llm import judge_with_llm | ||
|
|
||
| try: | ||
| result = judge_with_llm(commit_id, str(self.kernel_dir), str(self.target_project_dir)) | ||
| return result |
There was a problem hiding this comment.
judge_agent_llm uses a local-module import (from judge_llm import judge_with_llm). This will break when prejudge is imported/executed as a package (e.g., python -m prejudge.prejudge), where judge_llm won’t be on sys.path. Consider switching to from prejudge.judge_llm import judge_with_llm and ensuring the src directory is added to sys.path in the entrypoint (similar to src/prejudge/judge_llm.py), or convert the whole module to package-style imports consistently.
| _openrouter_common = partial( | ||
| ChatOpenAI, | ||
| temperature=0.0, | ||
| verbose=True, | ||
| base_url="https://newapi.sophie.pub/v1", | ||
| ) |
There was a problem hiding this comment.
The LLM client is configured with verbose=True and a hard-coded base_url at import time. This can unintentionally log prompt/patch contents and forces all runs to send code to a specific external endpoint. Please make verbosity default to False (or tie it to debug_mode) and make the endpoint configurable (e.g., via an env var) with a safe default.
| # Handle imports for both direct execution and module import | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| # Add src directory to path for imports | ||
| _src_path = Path(__file__).parent.parent | ||
| if str(_src_path) not in sys.path: | ||
| sys.path.insert(0, str(_src_path)) | ||
|
|
||
| from prejudge.judge_tools import create_locate_symbol_tool, create_view_code_tool | ||
| from prejudge.judge_prompt import JUDGE_SYSTEM_PROMPT, JUDGE_USER_PROMPT |
There was a problem hiding this comment.
Modifying sys.path at module import time is a global side effect that can mask packaging issues and change import resolution for the entire process. Since this file is now part of a package (src/prejudge/__init__.py), prefer proper package imports (and, if needed, adjust PYTHONPATH/entrypoints) rather than mutating sys.path inside the library module.
| # Handle imports for both direct execution and module import | |
| import sys | |
| from pathlib import Path | |
| # Add src directory to path for imports | |
| _src_path = Path(__file__).parent.parent | |
| if str(_src_path) not in sys.path: | |
| sys.path.insert(0, str(_src_path)) | |
| from prejudge.judge_tools import create_locate_symbol_tool, create_view_code_tool | |
| from prejudge.judge_prompt import JUDGE_SYSTEM_PROMPT, JUDGE_USER_PROMPT | |
| from .judge_tools import create_locate_symbol_tool, create_view_code_tool | |
| from .judge_prompt import JUDGE_SYSTEM_PROMPT, JUDGE_USER_PROMPT |
| if any( | ||
| word in response_lower | ||
| for word in ["vulnerability exists", "bug exists", "code is present", "found in"] | ||
| ): | ||
| return True |
There was a problem hiding this comment.
_parse_decision will return True for responses containing phrases like "not found in …" because the fallback tone check looks for the substring "found in" (and there is no corresponding negative indicator for "not found"). This mis-parses explicit negative answers and can inflate backport decisions. Add negative patterns like "not found"/"no occurrences" (or use a regex that excludes not found) and/or remove the ambiguous fallback substring check.
| from prejudge.judge_agent import JudgeAgent | ||
| from prejudge.prejudge import PrejudgeController | ||
|
|
||
| __all__ = ["JudgeAgent", "PrejudgeController"] No newline at end of file |
There was a problem hiding this comment.
Importing JudgeAgent and PrejudgeController at package import time makes import prejudge eagerly pull in LangChain/OpenAI dependencies and also triggers side effects from judge_agent.py (e.g., sys.path mutation). Consider avoiding eager imports here (e.g., expose names via lazy __getattr__, or only set __all__ and let callers import submodules explicitly) to keep package import lightweight and predictable.
| from prejudge.judge_agent import JudgeAgent | |
| from prejudge.prejudge import PrejudgeController | |
| __all__ = ["JudgeAgent", "PrejudgeController"] | |
| __all__ = ["JudgeAgent", "PrejudgeController"] | |
| def __getattr__(name): | |
| if name == "JudgeAgent": | |
| from prejudge.judge_agent import JudgeAgent | |
| return JudgeAgent | |
| if name == "PrejudgeController": | |
| from prejudge.prejudge import PrejudgeController | |
| return PrejudgeController | |
| raise AttributeError(f"module {__name__!r} has no attribute {name!r}") |
This PR is Part 2 of 2 for the AI-driven backport judgment feature.
Building upon the base tools introduced in Part 1, this PR implements the core LangChain agent engine and integrates it as the final check in our
prejudgepipeline flow.The agent analyzes whether the vulnerable code exists in the downstream target and employs a conservative strategy: it defaults to
YES(needs backport) when uncertain, ensuring no critical security patches are missed.