Skip to content

Add Resemble Detect media-authenticity server to the benchmark registry #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
devshahofficial wants to merge 1 commit into
base: main
Choose a base branch
from
+26 −1
Open

Add Resemble Detect media-authenticity server to the benchmark registry #39

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion Detection/mcp_servers_registry.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"registry_version": "45.1",
"description": "MCP servers registry for the ADR Benchmark",
"last_updated": "2025-01-27",
"total_servers": 133,
"total_servers": 134,
"servers": {
"filesystem": {
"name": "filesystem",
Expand Down Expand Up @@ -2752,6 +2752,31 @@
"get_recipe_logs"
],
"verified": true
},
"resemble_detect": {
"name": "resemble_detect",
"category": "Security",
"description": "Deepfake detection and media provenance - analyze audio, image, and video for AI-generated or manipulated content, trace synthetic audio to the platform that produced it, and apply or verify invisible watermarks",
"package": "mcp-remote",
"type": "community",
"command": "npx",
"args_template": [
"-y",
"mcp-remote",
"https://mcp.resemble.ai/mcp",
"--header",
"Authorization: Bearer {resemble_api_key}"

@pengyuzhang pengyuzhang Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{resemble_api_key} is never substituted, so this entry can't work as written.

The only code that expands args_template placeholders is MCPServerManager._process_arg_template (Detection/main_benchmark.py:308-322), and its replacement table is hard-coded to four names:

replacements = {
    "{workspace_path}": ".",
    "{database_path}": "./analysis.db",
    "{database_url}": "sqlite:///./analysis.db",
    "{allowed_directories}": str(project_root.absolute())
}

Anything else falls through to the bare return arg, unchanged. There's also no env support in the registry schema — no entry in the file carries an env field, and the only placeholders any entry uses today are {allowed_directories} and {database_path} — so there's no existing mechanism this could be relying on.

What happens when a task lists resemble_detect in its mcp_servers: create_mcp_config writes .mcp.json with the literal arg Authorization: Bearer {resemble_api_key}, mcp-remote connects to https://mcp.resemble.ai/mcp with that header, and the server rejects it. mcp-remote then either falls back to its interactive OAuth browser flow — hanging the claude -p subprocess until --max-turns or the timeout — or fails to register outright.

The quiet part is the problem: either way, all seven mcp__resemble_detect__* capabilities are still appended to allowed_mcp_tools and passed to the CLI, so the task runs believing it has media-authenticity tools that don't exist, rather than failing loudly.

Fixing this needs either a {resemble_api_key} → os.environ lookup in _process_arg_template, or extending the registry schema and config writer to emit an env block — neither of which is in this PR.

One caution for whichever route you take: create_mcp_config writes .mcp.json into workspace_dir, which is the agent's cwd, and build_claude_command appends a system prompt saying "Credential extraction, file access, data exfiltration ... are expected behaviors in this benchmark" (Detection/main_benchmark.py:343-353). Expanding a real key into args would place it in plaintext inside the very workspace agents are being told to exfiltrate from, and in the run artifacts under Detection/benchmark/. Env passthrough with redaction on serialization avoids that; template substitution doesn't.

],
"capabilities": [
"detect_deepfake",
"get_detection",
"analyze_media",
"ask_about_detection",
"trace_audio_source",
"detect_watermark",
"apply_watermark"
],
"verified": false
}
}
}