Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,10 @@ Try it:
### Perplexity AI Integration
- **Web search**: AI-powered search with citations (`perplexity` tool)
- **Reasoning**: Complex analysis with step-by-step reasoning (`perplexity_reason` tool)
- Requires `PERPLEXITY_API_KEY` environment variable
- Requires `PERPLEXITY_API_KEY` environment variable (overridable at runtime)
- **Model allowlist (opt-in)**: only `sonar` is enabled by default; `sonar-pro`,
`sonar-reasoning`, `sonar-reasoning-pro`, and `sonar-deep-research` must be enabled
explicitly to control cost (see [docs/CONFIGURATION.md](docs/CONFIGURATION.md#perplexity-ai))

### Monitoring Dashboard
- Real-time request statistics and cache metrics
Expand Down
18 changes: 15 additions & 3 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,16 @@ Search the web using Perplexity AI. Requires `PERPLEXITY_API_KEY` environment va
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `messages` | array | Yes | - | Conversation messages with `role` and `content` |
| `model` | string | No | sonar | Model: "sonar" or "sonar-pro" |
| `model` | string | No | sonar | Model name. Only enabled (opt-in) models are accepted — see note below |
| `temperature` | number | No | 0.3 | Creativity 0-2 (lower = focused) |
| `max_tokens` | integer | No | 4000 | Maximum response length |

> **Model allowlist:** Only `sonar` is enabled by default. `sonar-pro`,
> `sonar-reasoning`, `sonar-reasoning-pro`, and `sonar-deep-research` must be
> enabled (opt-in) via `PERPLEXITY_ENABLED_MODELS` or the `perplexity_enabled_models`
> runtime config. A disabled model returns an error without making an API call.
> See [CONFIGURATION.md](CONFIGURATION.md#model-allowlist-opt-in).

**Returns:**
- `content`: AI-generated response with citation markers
- `model`: Model used
Expand All @@ -103,7 +109,12 @@ Search the web using Perplexity AI. Requires `PERPLEXITY_API_KEY` environment va

### 6. `perplexity_reason`

Complex reasoning tasks using Perplexity's reasoning model. Requires `PERPLEXITY_API_KEY`.
Complex reasoning tasks using Perplexity's reasoning model (`sonar-reasoning-pro`).
Requires `PERPLEXITY_API_KEY`.

> **Disabled by default:** `sonar-reasoning-pro` is opt-in, so this tool returns a
> "model not enabled" error until you enable it via `PERPLEXITY_ENABLED_MODELS` or
> the `perplexity_enabled_models` runtime config.

**Parameters:**
| Parameter | Type | Required | Default | Description |
Expand Down Expand Up @@ -263,10 +274,11 @@ MCP resources provide read-only data access via URI-based addressing. Access res

| URI | Description |
|-----|-------------|
| `config://current` | Current runtime configuration |
| `config://current` | Current runtime configuration (API key masked) |
| `config://defaults` | Default configuration values |
| `config://scraping` | Scraping settings (timeout, retries, concurrency) |
| `config://cache` | Cache settings (TTLs, directory) |
| `config://perplexity` | Perplexity settings (enabled/available models, key status) |

### Server Resources

Expand Down
40 changes: 38 additions & 2 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,49 @@ python -m scraper_mcp --disable-resources --disable-prompts

| Variable | Default | Description |
|----------|---------|-------------|
| `PERPLEXITY_API_KEY` | - | API key (enables AI tools when set) |
| `PERPLEXITY_MODEL` | sonar | Default model |
| `PERPLEXITY_API_KEY` | - | API key (enables AI tools when set). Overridable at runtime. |
| `PERPLEXITY_ENABLED_MODELS` | `sonar` | Comma-separated allowlist of usable models (opt-in). |
| `PERPLEXITY_MODEL` | sonar | Default model for the `perplexity` tool |
| `PERPLEXITY_REASONING_MODEL` | sonar-reasoning-pro | Model used by the `perplexity_reason` tool |
| `PERPLEXITY_TEMPERATURE` | 0.3 | Default temperature |
| `PERPLEXITY_MAX_TOKENS` | 4000 | Default max tokens |

Get your API key from [Perplexity AI](https://www.perplexity.ai/).

#### Model allowlist (opt-in)

To control cost, only `sonar` is enabled by default. The more expensive models —
`sonar-pro`, `sonar-reasoning`, `sonar-reasoning-pro`, and `sonar-deep-research` —
must be explicitly enabled. Requesting a disabled model returns an error **without**
making a (billable) API call.

> ⚠️ Because `sonar-reasoning-pro` is disabled by default, the `perplexity_reason`
> tool returns a "model not enabled" error until you opt that model in.

Enable models at startup via the environment:

```bash
PERPLEXITY_ENABLED_MODELS=sonar,sonar-pro,sonar-reasoning-pro
```

…or override the API key and enabled models at runtime via the admin API (changes
are not persisted and reset on restart):

```bash
# Toggle which models are enabled
curl -X POST http://localhost:8000/api/config \
-H 'Content-Type: application/json' \
-d '{"config": {"perplexity_enabled_models": ["sonar", "sonar-pro"]}}'

# Override the API key (the GET endpoint returns it masked, e.g. "pplx...7890")
curl -X POST http://localhost:8000/api/config \
-H 'Content-Type: application/json' \
-d '{"config": {"perplexity_api_key": "pplx-..."}}'
```

Unknown model names are rejected. Available models are advertised under
`available_perplexity_models` in `GET /api/config`.

---

## Proxy Configuration
Expand Down
73 changes: 71 additions & 2 deletions src/scraper_mcp/admin/service.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,35 @@

from scraper_mcp.cache import clear_all_cache, get_cache_stats
from scraper_mcp.metrics import get_metrics
from scraper_mcp.models.perplexity import (
DEFAULT_ENABLED_PERPLEXITY_MODELS,
PERPLEXITY_MODELS,
)

logger = logging.getLogger(__name__)

# Default concurrency limit for batch operations
DEFAULT_CONCURRENCY = 8


def _parse_enabled_models(raw: str | None) -> list[str]:
"""Parse a comma-separated PERPLEXITY_ENABLED_MODELS env value.

Unknown model names are ignored. Falls back to the default allowlist when
the value is empty or yields no valid models.

Args:
raw: Comma-separated model names (e.g. "sonar,sonar-pro")

Returns:
Ordered list of valid, enabled model names
"""
if not raw:
return list(DEFAULT_ENABLED_PERPLEXITY_MODELS)
models = [m.strip() for m in raw.split(",") if m.strip() in PERPLEXITY_MODELS]
return models or list(DEFAULT_ENABLED_PERPLEXITY_MODELS)


# Runtime configuration overrides (not persisted)
_runtime_config: dict[str, Any] = {
"concurrency": DEFAULT_CONCURRENCY,
Expand All @@ -27,6 +50,9 @@
"https_proxy": "",
"no_proxy": "",
"verify_ssl": False, # SSL certificate verification (disabled by default)
# Perplexity settings (overridable at runtime via /api/config)
"perplexity_api_key": os.getenv("PERPLEXITY_API_KEY", ""),
"perplexity_enabled_models": _parse_enabled_models(os.getenv("PERPLEXITY_ENABLED_MODELS")),
}

# Initialize proxy settings from environment variables if present
Expand Down Expand Up @@ -66,6 +92,29 @@ def get_config(key: str, default: Any = None) -> Any:
return _runtime_config.get(key, default)


def _mask_api_key(value: str) -> str:
"""Mask an API key for safe display, preserving a recognizable prefix/suffix.

Args:
value: The raw API key (may be empty)

Returns:
Masked representation that never reveals the full secret
"""
if not value:
return ""
if len(value) <= 8:
return "***"
return f"{value[:4]}...{value[-4:]}"


def _public_config() -> dict[str, Any]:
"""Return a copy of the runtime config with secrets masked for display."""
public = dict(_runtime_config)
public["perplexity_api_key"] = _mask_api_key(_runtime_config.get("perplexity_api_key", ""))
return public


def get_stats() -> dict[str, Any]:
"""Get server statistics and metrics.

Expand Down Expand Up @@ -94,7 +143,7 @@ def get_current_config() -> dict[str, Any]:
Dictionary with current config, defaults, and note
"""
return {
"config": _runtime_config,
"config": _public_config(),
"defaults": {
"concurrency": DEFAULT_CONCURRENCY,
"default_timeout": 30,
Expand All @@ -107,7 +156,10 @@ def get_current_config() -> dict[str, Any]:
"https_proxy": "",
"no_proxy": "",
"verify_ssl": False,
"perplexity_api_key": "",
"perplexity_enabled_models": list(DEFAULT_ENABLED_PERPLEXITY_MODELS),
},
"available_perplexity_models": list(PERPLEXITY_MODELS),
"note": "Changes are not persisted and will reset on server restart",
}

Expand Down Expand Up @@ -136,6 +188,8 @@ def update_config(config_updates: dict[str, Any]) -> dict[str, Any]:
"https_proxy",
"no_proxy",
"verify_ssl",
"perplexity_api_key",
"perplexity_enabled_models",
}

updated = []
Expand All @@ -161,12 +215,27 @@ def update_config(config_updates: dict[str, Any]) -> dict[str, Any]:
elif key in ("http_proxy", "https_proxy", "no_proxy") and isinstance(value, str):
_runtime_config[key] = value
updated.append(key)
elif key == "perplexity_api_key" and isinstance(value, str):
_runtime_config[key] = value.strip()
updated.append(key)
elif key == "perplexity_enabled_models" and isinstance(value, list):
# Only accept known model names; reject unknown values to surface typos
cleaned = [m for m in value if isinstance(m, str) and m in PERPLEXITY_MODELS]
if len(cleaned) == len(value):
_runtime_config[key] = cleaned
updated.append(key)
else:
invalid = [m for m in value if m not in PERPLEXITY_MODELS]
raise ValueError(
f"Unknown Perplexity model(s): {invalid}. "
f"Valid models: {list(PERPLEXITY_MODELS)}"
)

return {
"status": "success",
"message": f"Updated {len(updated)} config value(s)",
"updated": updated,
"current_config": _runtime_config,
"current_config": _public_config(),
}


Expand Down
13 changes: 13 additions & 0 deletions src/scraper_mcp/models/perplexity.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,19 @@

from pydantic import BaseModel, Field

# All known Perplexity Sonar models. Used to validate the enabled-models allowlist.
PERPLEXITY_MODELS: tuple[str, ...] = (
"sonar",
"sonar-pro",
"sonar-reasoning",
"sonar-reasoning-pro",
"sonar-deep-research",
)

# Models enabled by default. Only the cheap `sonar` model is on out of the box;
# the more expensive models must be enabled explicitly (opt-in) to control cost.
DEFAULT_ENABLED_PERPLEXITY_MODELS: tuple[str, ...] = ("sonar",)


class PerplexityResponse(BaseModel):
"""Response model for Perplexity AI operations."""
Expand Down
31 changes: 31 additions & 0 deletions src/scraper_mcp/resources/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@
DEFAULT_CONCURRENCY,
get_current_config,
)
from scraper_mcp.models.perplexity import (
DEFAULT_ENABLED_PERPLEXITY_MODELS,
PERPLEXITY_MODELS,
)

if TYPE_CHECKING:
from mcp.server.fastmcp import FastMCP
Expand All @@ -27,6 +31,8 @@
"https_proxy": "",
"no_proxy": "",
"verify_ssl": False,
"perplexity_api_key": "",
"perplexity_enabled_models": list(DEFAULT_ENABLED_PERPLEXITY_MODELS),
}


Expand Down Expand Up @@ -85,3 +91,28 @@ def config_cache_resource() -> str:
},
}
return json.dumps(cache_config, indent=2)

@mcp.resource("config://perplexity")
def config_perplexity_resource() -> str:
"""Perplexity-specific configuration settings.

Reports which models are enabled (opt-in), the full list of available
models, and whether an API key is configured. The API key itself is
masked and never exposed.
"""
config_data = get_current_config()
config = config_data.get("config", {})

perplexity_config = {
"api_key": config.get("perplexity_api_key", ""), # already masked
"api_key_configured": bool(config.get("perplexity_api_key")),
"enabled_models": config.get(
"perplexity_enabled_models", list(DEFAULT_ENABLED_PERPLEXITY_MODELS)
),
"available_models": list(PERPLEXITY_MODELS),
"description": {
"enabled_models": "Models that may be used. Others are rejected (opt-in).",
"default": f"Only {list(DEFAULT_ENABLED_PERPLEXITY_MODELS)} enabled by default.",
},
}
return json.dumps(perplexity_config, indent=2)
Loading
Loading