Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pr-review-by-openhands.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
llm-base-url: https://llm-proxy.app.all-hands.dev
# [DEPRECATED] review-style is no longer used; standard and roasted are merged
# review-style: roasted
use-sub-agents: 'true' # Matches the default; kept for clarity
llm-api-key: ${{ secrets.LLM_API_KEY }}
github-token: ${{ secrets.OPENHANDS_BOT_GITHUB_PAT_PUBLIC }}
lmnr-api-key: ${{ secrets.LMNR_SKILLS_API_KEY }}
9 changes: 4 additions & 5 deletions plugins/pr-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Then configure the required secrets (see [Installation](#installation) below).
- **A/B Testing**: Support for testing multiple LLM models
- **Review Context Awareness**: Considers previous reviews and unresolved threads
- **Evidence Enforcement**: Optional check that PR descriptions include concrete end-to-end proof the code works, not just test output
- **Sub-Agent Delegation** *(Experimental)*: Split large PR reviews across multiple sub-agents, one per file, then consolidate findings (see [Known Limitations](#known-limitations-sub-agent-delegation))
- **Sub-Agent Delegation**: Split large PR reviews across multiple sub-agents, one per file, then consolidate findings (see [Known Limitations](#known-limitations-sub-agent-delegation))
- **Observability**: Optional Laminar integration for tracing and evaluation

## Plugin Contents
Expand Down Expand Up @@ -142,7 +142,7 @@ PR reviews are automatically triggered when:
| `llm-base-url` | No | `''` | Custom LLM endpoint URL |
| `review-style` | No | `roasted` | **[DEPRECATED]** Previously chose between `standard` and `roasted` review styles. Now ignored — the styles have been merged into a single unified skill. |
| `require-evidence` | No | `'false'` | Require the reviewer to enforce an `Evidence` section in the PR description with end-to-end proof: screenshots/videos for frontend work, commands and runtime output for backend or scripts, and an agent conversation link when applicable. Test output alone does not qualify. |
| `use-sub-agents` | No | `'false'` | **(Experimental)** Enable sub-agent delegation for file-level reviews. The main agent acts as a coordinator that delegates per-file review work to `file_reviewer` sub-agents via the SDK TaskToolSet, then consolidates findings into a single PR review. Useful for large PRs with many changed files. |
| `use-sub-agents` | No | `'true'` | Enable sub-agent delegation for file-level reviews. The main agent acts as a coordinator that delegates per-file review work to `file_reviewer` sub-agents via the SDK TaskToolSet, then consolidates findings into a single PR review. Useful for large PRs with many changed files. To restore the previous single-agent behavior, set to `'false'`. |
Comment thread
xingyaoww marked this conversation as resolved.
Comment thread
xingyaoww marked this conversation as resolved.
| `extensions-repo` | No | `OpenHands/extensions` | Extensions repository |
| `extensions-version` | No | `main` | Git ref (tag, branch, or SHA) |
| `llm-api-key` | Yes | - | LLM API key |
Expand All @@ -165,14 +165,13 @@ Python dependency caching is **disabled by default**. `uv run --with ...` re-dow

## Known Limitations: Sub-Agent Delegation

The `use-sub-agents` feature is **experimental** and has the following known constraints:
The following are known constraints of the sub-agent delegation feature. These are acceptable tradeoffs for the improved review depth it provides, and none pose a security risk — in the worst case a review may be less thorough than expected, which the single-agent fallback (`use-sub-agents: 'false'`) addresses.

- **LLM-driven JSON parsing**: The coordinator agent relies on the LLM to parse and merge JSON responses from sub-agents. There is no code-level validation of sub-agent output, so malformed responses may cause incomplete reviews.
- **Potential information loss during consolidation**: When merging findings from multiple sub-agents, the coordinator may lose or deduplicate findings imperfectly, especially for cross-file issues.
- **No integration tests yet**: Current test coverage verifies prompt formatting only. End-to-end validation of the delegation flow requires manual workflow testing.
- **Sub-agents have read-only tools**: File reviewer sub-agents have access to `terminal` and `file_editor` for inspecting full source files and surrounding context, but they cannot query the GitHub API or post reviews — only the coordinator handles GitHub interaction.
Comment thread
xingyaoww marked this conversation as resolved.
Comment thread
xingyaoww marked this conversation as resolved.

These limitations are acceptable for an opt-in experimental feature and will be addressed as the feature matures.
To opt out, set `use-sub-agents: 'false'` in your workflow.

## A/B Testing Multiple Models

Expand Down
5 changes: 3 additions & 2 deletions plugins/pr-review/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,12 @@ inputs:
default: 'false'
use-sub-agents:
description: >
Enable sub-agent delegation for file-level reviews (experimental).
Enable sub-agent delegation for file-level reviews.
When true, the agent gets the TaskToolSet and decides at runtime
whether to delegate based on diff size and complexity.
Set to 'false' to restore the previous single-agent behavior.
required: false
default: 'false'
default: 'true'
Comment thread
xingyaoww marked this conversation as resolved.
Comment thread
xingyaoww marked this conversation as resolved.
extensions-repo:
description: GitHub repository for extensions (owner/repo)
required: false
Expand Down
1 change: 1 addition & 0 deletions plugins/pr-review/workflows/pr-review-by-openhands.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
llm-base-url: https://llm-proxy.app.all-hands.dev
# [DEPRECATED] review-style is no longer used; standard and roasted are merged
# review-style: roasted
use-sub-agents: 'true' # Matches the default; kept for clarity
Comment thread
xingyaoww marked this conversation as resolved.
llm-api-key: ${{ secrets.LLM_API_KEY }}
github-token: ${{ secrets.OPENHANDS_BOT_GITHUB_PAT_PUBLIC }}
lmnr-api-key: ${{ secrets.LMNR_SKILLS_API_KEY }}
Loading