Skip to content

feat: integrate LangChain agent into prejudge pipeline#13

Merged
LeiGuo622 merged 1 commit into
mainfrom
feat/agent-pipeline
Apr 13, 2026
Merged

feat: integrate LangChain agent into prejudge pipeline#13
LeiGuo622 merged 1 commit into
mainfrom
feat/agent-pipeline

Conversation

@LeiGuo622
Copy link
Copy Markdown
Owner

This PR is Part 2 of 2 for the AI-driven backport judgment feature.
Building upon the base tools introduced in Part 1, this PR implements the core LangChain agent engine and integrates it as the final check in our prejudge pipeline flow.
The agent analyzes whether the vulnerable code exists in the downstream target and employs a conservative strategy: it defaults to YES (needs backport) when uncertain, ensuring no critical security patches are missed.

Signed-off-by: LeiGuo622 <2931505649@qq.com>
Copilot AI review requested due to automatic review settings April 13, 2026 12:44
@LeiGuo622 LeiGuo622 merged commit a81aef3 into main Apr 13, 2026
3 checks passed
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR integrates an LLM (LangChain) “judge agent” into the prejudge pipeline as the final decision step for determining whether a patch needs backporting by checking if vulnerable code exists in the downstream target.

Changes:

  • Added an LLM-based final check (judge_agent_llm) to PrejudgeController.analyze_and_report.
  • Introduced a new JudgeAgent LangChain executor with tools/prompting and decision parsing.
  • Added prejudge package initializer exporting JudgeAgent and PrejudgeController.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
src/prejudge/prejudge.py Adds a new LLM-agent judgment step to produce the final true/false output.
src/prejudge/judge_agent.py Implements the LangChain-based agent, model configuration, patch retrieval, and response parsing.
src/prejudge/__init__.py Declares prejudge as a package and re-exports key entry points.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/prejudge/prejudge.py
Comment on lines +97 to +101
from judge_llm import judge_with_llm

try:
result = judge_with_llm(commit_id, str(self.kernel_dir), str(self.target_project_dir))
return result
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

judge_agent_llm uses a local-module import (from judge_llm import judge_with_llm). This will break when prejudge is imported/executed as a package (e.g., python -m prejudge.prejudge), where judge_llm won’t be on sys.path. Consider switching to from prejudge.judge_llm import judge_with_llm and ensuring the src directory is added to sys.path in the entrypoint (similar to src/prejudge/judge_llm.py), or convert the whole module to package-style imports consistently.

Copilot uses AI. Check for mistakes.
Comment on lines +35 to +40
_openrouter_common = partial(
ChatOpenAI,
temperature=0.0,
verbose=True,
base_url="https://newapi.sophie.pub/v1",
)
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LLM client is configured with verbose=True and a hard-coded base_url at import time. This can unintentionally log prompt/patch contents and forces all runs to send code to a specific external endpoint. Please make verbosity default to False (or tie it to debug_mode) and make the endpoint configurable (e.g., via an env var) with a safe default.

Copilot uses AI. Check for mistakes.
Comment on lines +20 to +30
# Handle imports for both direct execution and module import
import sys
from pathlib import Path

# Add src directory to path for imports
_src_path = Path(__file__).parent.parent
if str(_src_path) not in sys.path:
sys.path.insert(0, str(_src_path))

from prejudge.judge_tools import create_locate_symbol_tool, create_view_code_tool
from prejudge.judge_prompt import JUDGE_SYSTEM_PROMPT, JUDGE_USER_PROMPT
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modifying sys.path at module import time is a global side effect that can mask packaging issues and change import resolution for the entire process. Since this file is now part of a package (src/prejudge/__init__.py), prefer proper package imports (and, if needed, adjust PYTHONPATH/entrypoints) rather than mutating sys.path inside the library module.

Suggested change
# Handle imports for both direct execution and module import
import sys
from pathlib import Path
# Add src directory to path for imports
_src_path = Path(__file__).parent.parent
if str(_src_path) not in sys.path:
sys.path.insert(0, str(_src_path))
from prejudge.judge_tools import create_locate_symbol_tool, create_view_code_tool
from prejudge.judge_prompt import JUDGE_SYSTEM_PROMPT, JUDGE_USER_PROMPT
from .judge_tools import create_locate_symbol_tool, create_view_code_tool
from .judge_prompt import JUDGE_SYSTEM_PROMPT, JUDGE_USER_PROMPT

Copilot uses AI. Check for mistakes.
Comment on lines +269 to +273
if any(
word in response_lower
for word in ["vulnerability exists", "bug exists", "code is present", "found in"]
):
return True
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_parse_decision will return True for responses containing phrases like "not found in …" because the fallback tone check looks for the substring "found in" (and there is no corresponding negative indicator for "not found"). This mis-parses explicit negative answers and can inflate backport decisions. Add negative patterns like "not found"/"no occurrences" (or use a regex that excludes not found) and/or remove the ambiguous fallback substring check.

Copilot uses AI. Check for mistakes.
Comment thread src/prejudge/__init__.py
Comment on lines +7 to +10
from prejudge.judge_agent import JudgeAgent
from prejudge.prejudge import PrejudgeController

__all__ = ["JudgeAgent", "PrejudgeController"] No newline at end of file
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Importing JudgeAgent and PrejudgeController at package import time makes import prejudge eagerly pull in LangChain/OpenAI dependencies and also triggers side effects from judge_agent.py (e.g., sys.path mutation). Consider avoiding eager imports here (e.g., expose names via lazy __getattr__, or only set __all__ and let callers import submodules explicitly) to keep package import lightweight and predictable.

Suggested change
from prejudge.judge_agent import JudgeAgent
from prejudge.prejudge import PrejudgeController
__all__ = ["JudgeAgent", "PrejudgeController"]
__all__ = ["JudgeAgent", "PrejudgeController"]
def __getattr__(name):
if name == "JudgeAgent":
from prejudge.judge_agent import JudgeAgent
return JudgeAgent
if name == "PrejudgeController":
from prejudge.prejudge import PrejudgeController
return PrejudgeController
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Copilot uses AI. Check for mistakes.
@LeiGuo622 LeiGuo622 deleted the feat/agent-pipeline branch April 15, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants