SDK vs Agent Server: Close compatibility gaps identified in compatibility matrix

[xingyaoww]

Context

A compatibility matrix comparing the Python SDK (openhands-sdk) vs Agent Server (openhands-agent-server) feature support has been published:

🔗 Compatibility Matrix

Overall the compatibility looks pretty solid, but there are several areas where the agent-server lacks parity with the SDK that would be valuable to close.

Key gaps to address

The following features are supported in the SDK but not yet exposed (or only partially exposed) in the agent-server:

High Priority

Feature	SDK	Server	Notes
Sub-agents & delegation	✔	✘	SDK-only: register_agent, load_agents_from_dir. Vasco is actively working on this — believed to be close.
Critic / iterative refinement	✔	✘	SDK-only: CriticBase, APIBasedCritic, etc.
Multi-LLM / LLM fallback strategies	✔	✘	Automatic model fallback on failure — SDK only
LLM completions (direct)	✔	◐	Server uses LLM internally but has no direct completion endpoint; exposed indirectly via conversations

Additional gaps

Feature	SDK	Server	Notes
Delegate tool (sub-agents)	✔	✘	SDK-only delegation to sub-agents
Custom tool registration	✔	✘	SDK: @register_tool decorator for custom tools
MCP client (direct API)	✔	✘	SDK-only: MCPClient for programmatic MCP access
Cached git repo	✔	✘	SDK-only: CachedRepo for efficient git ops
Plugin system (install/uninstall/manage)	✔	✘	SDK-only: full plugin lifecycle management
Plugin marketplace	✔	✘	SDK-only: Marketplace, MarketplaceEntry
Token usage tracking	✔	◐	Tracked internally; exposed via conversation metrics
Stuck detector	✔	◐	Runs inside server conversations automatically
Hook execution engine	✔	◐	Executor runs inside conversations; no direct API
Security risk assessment	✔	◐	Runs inline; not a standalone endpoint

Suggested addition (from discussion)

• LLMProfileStore — gives the agent-server the ability to load and persist LLM profiles across sessions

Future plans

• Add CLI/web and possibly the TypeScript client to the compatibility matrix
• Set up a CI job to keep the matrix updated automatically

---

Related PR: #2710

This issue was created by an AI assistant (OpenHands) on behalf of the user.

[VascoSch92]

Architecture note

The agent-server is intentionally not a CRUD settings store. The split:

• SDK (openhands-sdk/openhands/sdk/settings/model.py) — defines LLMAgentSettings, ACPAgentSettings, ConversationSettings, VerificationSettings, CondenserSettings. Full config tree including
  critic, security-analyzer, hooks, MCP, tools, plugins.
• Agent-server (openhands-agent-server/openhands/agent_server/settings_router.py) — exposes the schema only: GET /settings/agent-schema and GET /settings/conversation-schema. Settings are applied
  per-conversation via StartConversationRequest.
• OpenHands/OpenHands (openhands/app_server/settings/settings_router.py) — full CRUD store and LLM profile store: GET/POST /settings, GET/POST/DELETE /settings/profiles/{name}, plus activate,
  rename.

Many "gaps" are already covered because the AgentSettings / ConversationSettings fields flow into the conversation at startup — no dedicated REST route needed.

Status per gap

#	Feature	Status
1	Sub-agents & delegation	DONE — via #2710
2	Delegate tool	DONE — deprecated, superseded by TaskToolSet
3	Critic / iterative refinement	DONE via settings
4	Multi-LLM / fallback	PARTIAL
5	LLM completions direct endpoint	PARTIAL — covered by ask_agent for conversation-bound Q&A, no stateless route
6	Custom tool registration	DONE via settings
7	MCP client direct API	DONE via settings
8	Cached git repo (CachedRepo)	N/A — class not present in current codebase
9	Plugin system	PARTIAL
10	Plugin marketplace	NOT DONE
11	Token usage tracking	PARTIAL
12	Stuck detector	DONE via settings
13	Hook execution engine	DONE via settings
14	Security risk assessment	DONE
15	LLMProfileStore (suggested)	DONE upstream — in OpenHands/OpenHands, out-of-scope for agent-server

Evidence per gap

1. Sub-agents & delegation — DONE

• SDK: openhands-sdk/openhands/sdk/subagent/registry.py:119 (register_agent)
• Server: openhands-agent-server/openhands/agent_server/tool_router.py:16 (register_builtins_agents on startup)
• Per-conversation via ConversationSettings.agent_definitions

2. Delegate tool — DONE (superseded)

• SDK: openhands-tools/openhands/tools/delegate/definition.py:15 — deprecated in v1.16.0, removal v1.23.0
• Replaced by TaskToolSet, auto-registered in tool_router.py

3. Critic / iterative refinement — DONE via settings

• SDK: openhands-sdk/openhands/sdk/critic/base.py:18, critic/impl/api/critic.py:27
• Configured via LLMAgentSettings.verification.* fields: critic_enabled, critic_mode, critic_threshold, critic_server_url, critic_model_name, enable_iterative_refinement,
  max_refinement_iterations
• LLMAgentSettings.build_critic() auto-applies at agent creation
• Exposed in GET /settings/agent-schema

4. Multi-LLM / fallback — PARTIAL

• SDK: openhands-sdk/openhands/sdk/llm/fallback_strategy.py:39
• Not surfaced in LLMAgentSettings
• Profile switching handled upstream (OpenHands/OpenHands profile store); open: design(agent-server): REST LLM profiles for conversation start and restore #2485, feat(agent-server): add profile-based LLM switching for conversations #2576

5. LLM completions direct endpoint — PARTIAL

• POST /conversations/{id}/ask_agent (conversation_router.py:368) covers conversation-bound Q&A
• ask_agent injects conversation history and wraps the question in a template — no arbitrary-messages/streaming/tool_choice path
• A stateless /llm/completions route would serve eval harnesses, title gen, classification

6. Custom tool registration — DONE via settings

• SDK: openhands-sdk/openhands/sdk/tool/registry.py:119 (register_tool decorator)
• Per-conversation via LLMAgentSettings.tools
• GET /tools in tool_router.py:21 lists available

7. MCP client direct API — DONE via settings

• SDK: openhands-sdk/openhands/sdk/mcp/client.py:10
• Per-conversation via LLMAgentSettings.mcp_config
• No standalone /mcp/* route — may be worth adding if callers need direct MCP access outside a conversation

8. Cached git repo — N/A

• No CachedRepo class in current SDK; matrix entry likely stale

9. Plugin system — PARTIAL

• SDK: openhands-sdk/openhands/sdk/plugin/, PluginSource
• Per-conversation via ConversationSettings.plugins
• skills_router.py covers skills; no plugin install/uninstall REST endpoints

10. Plugin marketplace — NOT DONE

• SDK: openhands-sdk/openhands/sdk/marketplace/types.py:20 (MarketplaceEntry)
• No server route; feat(plugin): Support multiple marketplace registrations with auto-load semantics #2495 open

11. Token usage tracking — PARTIAL

• SDK: openhands-sdk/openhands/sdk/llm/metrics.py
• Available via conversation info / event stream; logged in event_service.py:404
• No dedicated /token-usage endpoint

12. Stuck detector — DONE via settings

• SDK: openhands-sdk/openhands/sdk/conversation/stuck_detector.py:22
• Auto-runs inside conversation; toggled via ConversationSettings.stuck_detection

13. Hook execution engine — DONE via settings

• SDK: openhands-sdk/openhands/sdk/hooks/
• Per-conversation via ConversationSettings.hook_config
• GET /hooks exposes configured hooks

14. Security risk assessment — DONE

• SDK: openhands-sdk/openhands/sdk/security/risk.py, security/llm_analyzer.py
• Persisted via ConversationSettings.security_analyzer (default "llm")
• Runtime update: PATCH /conversations/{id}/security-analyzer (conversation_router.py:279)

15. LLMProfileStore — DONE upstream

• SDK: openhands-sdk/openhands/sdk/llm/llm_profile_store.py:18
• Full CRUD in OpenHands/OpenHands at openhands/app_server/settings/settings_router.py:
  • GET /profiles
  • GET /profiles/{name}
  • POST /profiles/{name} (create/update)
  • DELETE /profiles/{name}
  • POST /profiles/{name}/activate
  • POST /profiles/{name}/rename
• Intentionally out-of-scope for agent-server

Related merged PRs (beyond #2710)

• feat(agent-server): expose settings schema #2651 — expose settings schema from agent-server (enables gaps 3, 6, 7, 9, 12, 13, 14)
• Move marketplace definitions to openhands.sdk.marketplace module #2786 — move marketplace definitions into openhands.sdk.marketplace
• feat(pr-review): enable sub-agent delegation for file-level reviews #2839 — sub-agent delegation for PR reviews
• feat(sdk/subagent): Support LLM profiles in subagents  #2258 / feat(sdk/subagent): add profile_store_dir #2340 — LLM profiles in subagents, profile_store_dir
• feat: add title_llm_profile support for auto-titling #2515 — title_llm_profile for auto-titling
• feat: Extensions module #2801 — Extensions module
• fix(tools): prevent duplicate WithRisk/WithSummary Action classes in sub-agent conversations #2649 — fix duplicate Action classes in sub-agent conversations

Open PRs addressing remaining gaps

• design(agent-server): REST LLM profiles for conversation start and restore #2485 — REST LLM profiles for conversation start/restore (gap 4/15 wiring)
• feat(agent-server): add profile-based LLM switching for conversations #2576 — profile-based LLM switching for conversations (gap 4)
• feat(plugin): Support multiple marketplace registrations with auto-load semantics #2495 — multi-marketplace registrations (gap 10)
• Enable delegation by default in agent-server #2691 — enable delegation by default (gap 1 polish)
• feat(skills): add context=fork for isolated subagent execution (1/2) #2823 — context=fork for isolated subagent execution (gap 1)
• settings: move critic refinement ownership into SDK #2620 — move critic refinement ownership into SDK (gap 3)

Bottom line

Of 15 items: 9 effectively DONE (5 fully, 4 via the AgentSettings / ConversationSettings schema), 3 PARTIAL (gaps 4, 5, 11), 1 NOT DONE (gap 10, in flight via #2495), 1 N/A (gap 8), 1
upstream (gap 15). Remaining server-side work worth considering:

1. Stateless /llm/completions route (gap 5) — for callers that want raw LLM access without a conversation
2. Marketplace REST endpoints (gap 10) — feat(plugin): Support multiple marketplace registrations with auto-load semantics #2495 in flight
3. Dedicated token-usage endpoint (gap 11) — if conversation metrics aren't enough
4. Fallback strategy in LLMAgentSettings (gap 4) — partially covered by design(agent-server): REST LLM profiles for conversation start and restore #2485/feat(agent-server): add profile-based LLM switching for conversations #2576

[enyst]

To note, I think maybe this :

The agent-server is intentionally not a CRUD settings store. The split:
...
OpenHands/OpenHands (openhands/app_server/settings/settings_router.py) — full CRUD store and LLM profile store: GET/POST /settings, GET/POST/DELETE /settings/profiles/{name}, plus activate,
rename.

🤔 doesn't depend on the Web OH/OH. I think maybe it needs CRUD on profiles, e.g. slack. The discussion digressed, but the point is that maybe they do belong there, because it enables easy use of the agent-server with various clients that people may make.

[VascoSch92]

Status update

Changes since the previous status comment:

• Gap 3 (Critic / iterative refinement) — now fully done; settings: move critic refinement ownership into SDK #2620 merged 2026-05-07 moves critic refinement ownership into the SDK, on top of the existing settings-based wiring.
• Gap 4 (Multi-LLM / fallback) — both PRs that were in flight, design(agent-server): REST LLM profiles for conversation start and restore #2485 (design: REST LLM profiles) and feat(agent-server): add profile-based LLM switching for conversations #2576 (profile-based LLM switching), were closed without merging on 2026-05-08. This gap currently has no active server-side work. Fallback strategy is still not surfaced in LLMAgentSettings.
• Gap 10 (Plugin marketplace) — feat(plugin): Support multiple marketplace registrations with auto-load semantics #2495 still open.
• Gap 1 polish — Enable delegation by default in agent-server #2691 (enable delegation by default) and feat(skills): add context=fork for isolated subagent execution (1/2) #2823 (context=fork for isolated subagent execution) still open.

Updated remaining server-side work

1. Multi-LLM / fallback (gap 4) — needs a fresh approach now that design(agent-server): REST LLM profiles for conversation start and restore #2485/feat(agent-server): add profile-based LLM switching for conversations #2576 are closed. Worth a short design discussion before reopening.
2. Stateless /llm/completions route (gap 5) — unchanged.
3. Marketplace REST endpoints (gap 10) — feat(plugin): Support multiple marketplace registrations with auto-load semantics #2495 in flight.
4. Dedicated token-usage endpoint (gap 11) — unchanged.

Re: @enyst's point on profile CRUD

Good point — the agent-server should be usable by clients other than OpenHands/OpenHands, so having profile CRUD live only upstream forces every client to reimplement it. I'd lean toward adding a minimal profile CRUD surface to the agent-server (backed by LLMProfileStore) and letting OpenHands/OpenHands continue to layer its richer settings store on top. Happy to spin this out into its own issue if that's the direction.