diff --git a/.github/workflows/gitleaks.yml b/.github/workflows/gitleaks.yml new file mode 100644 index 0000000..34752ab --- /dev/null +++ b/.github/workflows/gitleaks.yml @@ -0,0 +1,35 @@ +name: Gitleaks + +on: + push: + pull_request: + +permissions: + contents: read + +jobs: + scan: + name: Scan Git history + runs-on: ubuntu-latest + steps: + - name: Check out repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Install and run gitleaks + shell: bash + run: | + set -euo pipefail + version=8.30.1 + archive="gitleaks_${version}_linux_x64.tar.gz" + download="https://github.com/gitleaks/gitleaks/releases/download/v${version}/${archive}" + + curl --fail --location --retry 5 --retry-all-errors \ + --output "${RUNNER_TEMP}/${archive}" "${download}" + echo "551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb ${RUNNER_TEMP}/${archive}" \ + | sha256sum --check --strict + tar --extract --gzip --file "${RUNNER_TEMP}/${archive}" \ + --directory "${RUNNER_TEMP}" gitleaks + + "${RUNNER_TEMP}/gitleaks" git --no-banner --redact . diff --git a/README.md b/README.md index ab76a6f..8c027d5 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ ## News -- **[2026-08-08] 🚀 Unified LiteLLM gateway.** SkillHone now accepts one +- **[2026-08-08] Unified LiteLLM gateway.** SkillHone now accepts one `provider/model` configuration across Anthropic, DeepSeek, OpenAI, Gemini, and other LiteLLM providers—no external Anthropic-compatible endpoint or transport switch required. Improver, executor, and synthesis can each use @@ -247,30 +247,27 @@ assistant these values when you install — it will write the right | Role | Required? | What it does | |---|---|---| -| **Optimizer** | required | Drives the optimisation loop — proposes patches to the skill. | -| **Executor** | optional, defaults to Optimizer | Runs the skill being tested on each probe. | -| **Tester** | optional, defaults to Optimizer | Scores / judges the executor's output. | - -If you use Anthropic directly, just give the assistant your -Anthropic API key — `claude-agent-sdk` uses Anthropic's official -endpoint by default. **Only when you route through a third-party -Anthropic-compatible provider** (e.g. DeepSeek) do you need to -fill the three fields per role: `base_url` (Anthropic-format), -`api_key`, `model_name`. Example: - -```ini -base_url = https://api.deepseek.com/anthropic -api_key = sk-xxx -model_name = deepseek-v4-pro -``` +| **Improver** | required | Drives the optimisation loop — proposes patches to the skill. | +| **Executor** | optional, defaults to Improver | Runs the skill being tested on each probe. | +| **Synthesis** | optional, defaults to Improver | Generates evaluation data when using the synthesis workflow. | + +For each role, provide an `api_key` and a LiteLLM `provider/model` name. Add +`api_base` only when the provider needs a non-default upstream endpoint. Each +role can use independent credentials and endpoints; omitted optional roles +reuse the Improver profile. -Model profiles use LiteLLM's `provider/model` naming. SkillHone starts a -private local bridge and supplies its Anthropic Messages endpoint to Claude -Agent SDK automatically—no transport switch or external Anthropic-compatible -endpoint is needed. This also covers Anthropic models (`anthropic/claude-…`): +SkillHone starts a loopback-only LiteLLM bridge and supplies its Anthropic +Messages endpoint to Claude Agent SDK automatically. You do not need to run a +proxy or find an Anthropic-compatible endpoint. The same configuration covers +DeepSeek, Anthropic, OpenAI, Gemini, and other LiteLLM providers: ```jsonc -{"improver":{"model":"deepseek/deepseek-chat","api_key":"sk-xxx","api_base":"https://api.deepseek.com/v1"}} +{ + "improver": { + "model": "deepseek/deepseek-chat", + "api_key": "sk-xxx" + } +} ``` Full schema, multi-identity Forgejo tokens, and the `~/.skillhone/` diff --git a/docs/README.zh.md b/docs/README.zh.md index 7368575..f553956 100644 --- a/docs/README.zh.md +++ b/docs/README.zh.md @@ -50,7 +50,7 @@ ## 最新动态 -- **[2026-08-08] 🚀 统一 LiteLLM 模型网关。** SkillHone 现已使用一套 +- **[2026-08-08] 统一 LiteLLM 模型网关。** SkillHone 现已使用一套 `provider/model` 配置连接 Anthropic、DeepSeek、OpenAI、Gemini 及其他 LiteLLM provider，无需外部 Anthropic-compatible endpoint，也无需切换 transport。Improver、Executor 和 Synthesis 可分别使用独立凭据与上游 @@ -229,29 +229,26 @@ AI 助手就行，剩下的 `~/.skillhone/settings.json` 它会自己写。 | 角色 | 是否必填 | 作用 | |---|---|---| -| **Optimizer** | 必填 | 驱动优化闭环 —— 给技能写补丁。 | -| **Executor** | 可选，不填则与 Optimizer 一致 | 在每条 probe 上运行被测技能。 | -| **Tester** | 可选，不填则与 Optimizer 一致 | 给执行结果打分 / 判定。 | - -如果直接走 Anthropic 官方,只需要给 AI 助手一个 Anthropic API -key 即可 —— `claude-agent-sdk` 默认就走 Anthropic 官方 endpoint。 -**只有在你需要把流量转到第三方 Anthropic 兼容代理时**(例如 -DeepSeek),才需要每个角色填三个字段:`base_url`(Anthropic -格式)、`api_key`、`model_name`。例如: - -```ini -base_url = https://api.deepseek.com/anthropic -api_key = sk-xxx -model_name = deepseek-v4-pro -``` +| **Improver** | 必填 | 驱动优化闭环 —— 给技能写补丁。 | +| **Executor** | 可选，不填则与 Improver 一致 | 在每条 probe 上运行被测技能。 | +| **Synthesis** | 可选，不填则与 Improver 一致 | 在使用数据合成工作流时生成评测数据。 | + +每个角色只需提供 `api_key` 和 LiteLLM 的 `provider/model` 名称；仅当 +provider 需要非默认上游地址时才填写 `api_base`。不同角色可以使用各自 +独立的凭据和上游地址，省略的可选角色会复用 Improver 配置。 -模型统一使用 LiteLLM 的 `provider/model` 名称，不再配置 transport。 -SkillHone 会自动启动仅监听本机的转换代理， -并把 Anthropic Messages 地址注入 Claude Agent SDK，无需用户自行部署 -Anthropic-compatible endpoint： +SkillHone 会自动启动仅监听本机的 LiteLLM 转换代理，并把 Anthropic +Messages 地址注入 Claude Agent SDK。用户无需自行运行代理，也无需寻找 +Anthropic-compatible endpoint。同一套配置适用于 DeepSeek、Anthropic、 +OpenAI、Gemini 及其他 LiteLLM provider： ```jsonc -{"improver":{"model":"deepseek/deepseek-chat","api_key":"sk-xxx","api_base":"https://api.deepseek.com/v1"}} +{ + "improver": { + "model": "deepseek/deepseek-chat", + "api_key": "sk-xxx" + } +} ``` 完整字段、多身份 Forgejo token、以及 `~/.skillhone/` 下的目录结构都在 diff --git a/docs/install/developer.md b/docs/install/developer.md index 5008fe2..207a1c7 100644 --- a/docs/install/developer.md +++ b/docs/install/developer.md @@ -96,7 +96,8 @@ attributes. Symlinks also work on regular filesystems. ## 4. Write `~/.skillhone/settings.json` -Save this template, filling in the four bracketed values. The full schema lives +Save this template with a LiteLLM `provider/model` name and credentials for +each role. The full schema lives at [`skills/skillhone/references/configuration.md`](../../skills/skillhone/references/configuration.md). ```bash @@ -104,8 +105,6 @@ mkdir -p ~/.skillhone cat > ~/.skillhone/settings.json <<'EOF' { - "api_key": "", - "forgejo": { "url": "http://localhost:3000", "owner": "skillhone", @@ -113,50 +112,32 @@ cat > ~/.skillhone/settings.json <<'EOF' }, "improver": { - "api_base": "", - "model": "", + "api_key": "", + "model": "", + "api_base": "", "sdk_model_alias": "opus", "max_turns": 100, - "env": { - "ANTHROPIC_BASE_URL": "", - "ANTHROPIC_API_KEY": "", - "ANTHROPIC_MODEL": "", - "ANTHROPIC_DEFAULT_OPUS_MODEL": "", - "ANTHROPIC_DEFAULT_SONNET_MODEL": "", - "ANTHROPIC_AUTH_TOKEN": "", - "ANTHROPIC_CUSTOM_HEADERS": "" - } + "env": {} }, "executor": { - "api_base": "", - "model": "", + "api_key": "", + "model": "", + "api_base": "", "sdk_model_alias": "haiku", "workers": 2, "max_iterations": 150, "thinking_enabled": true, "context_size": 40000, - "env": { - "ANTHROPIC_BASE_URL": "", - "ANTHROPIC_API_KEY": "", - "ANTHROPIC_MODEL": "", - "ANTHROPIC_DEFAULT_HAIKU_MODEL": "", - "ANTHROPIC_AUTH_TOKEN": "", - "ANTHROPIC_CUSTOM_HEADERS": "" - } + "env": {} }, "synthesis": { - "api_base": "", - "model": "", + "api_key": "", + "model": "", + "api_base": "", "workers": 2, - "env": { - "ANTHROPIC_BASE_URL": "", - "ANTHROPIC_API_KEY": "", - "ANTHROPIC_MODEL": "", - "ANTHROPIC_AUTH_TOKEN": "", - "ANTHROPIC_CUSTOM_HEADERS": "" - } + "env": {} } } EOF @@ -164,18 +145,15 @@ EOF chmod 600 ~/.skillhone/settings.json ``` -Two things worth knowing: - -- **`ANTHROPIC_BASE_URL` is the prefix only.** The SDK appends `/v1/messages` - itself; strip any `/v1/messages` suffix your provider's docs hand you. -- **The blank `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_CUSTOM_HEADERS` is - deliberate.** They shadow any inherited shell env so the SDK uses - `ANTHROPIC_API_KEY` instead of a stale token from the parent shell — a - silent source of `401`s otherwise. +`api_base` is optional: omit it to use LiteLLM's standard endpoint for the +provider. For example, DeepSeek can use `deepseek/deepseek-chat` with no +`api_base`; Anthropic can use an `anthropic/claude-...` model. SkillHone starts +a loopback-only proxy and injects its Anthropic Messages endpoint into Claude +Agent SDK automatically. Do not configure `ANTHROPIC_BASE_URL` yourself. -If you talk to Anthropic directly, set `api_base` to `https://api.anthropic.com` -and the model to a Claude model name. For DeepSeek's Anthropic-compatible -endpoint, use `https://api.deepseek.com/anthropic` and `deepseek-v4-pro` / etc. +The Executor and Synthesis sections are optional. Omit either section to reuse +the Improver model profile. Prefer `api_key_env` over `api_key` when credentials +are already managed in the process environment. --- diff --git a/docs/install/skillhone.md b/docs/install/skillhone.md index 49d93dc..868303a 100644 --- a/docs/install/skillhone.md +++ b/docs/install/skillhone.md @@ -78,30 +78,17 @@ SkillHone reads `~/.skillhone/settings.json` for its model credentials. The only thing you actually need from the user is **one set of model credentials** for the optimisation loop. -**Default — Anthropic direct.** Ask only for an Anthropic API key. -`claude-agent-sdk` uses Anthropic's official endpoint by default, so -no `base_url` / `model_name` overrides are needed. - -**Third-party Anthropic-compatible providers** (e.g. DeepSeek). -Only when the user explicitly routes through such a -provider, ask for three fields: +Ask for the provider API key and a LiteLLM `provider/model` identifier. Add an +upstream `api_base` only when the provider does not use LiteLLM's default +endpoint: | Required | Optional | |---|---| -| **Optimizer LLM** — `base_url` (Anthropic-format), `api_key`, `model_name`. | **Executor LLM** — same three fields. If omitted, SkillHone reuses the optimizer settings for the eval solver. | - -Example values to ask for, in plain English: - -``` -base_url = https://api.deepseek.com/anthropic -api_key = sk-xxx -model_name = deepseek-v4-pro -``` +| **Improver** — `api_key`, `model` (`provider/model`). | **Executor** and **Synthesis** — independent `api_key`, `model`, and optional `api_base`. Omitted roles reuse the Improver profile. | -**Unified LiteLLM adapter (no Anthropic-compatible endpoint required).** Ask -for the provider API key and a LiteLLM `provider/model` identifier. SkillHone -starts and stops a loopback-only proxy automatically. Anthropic uses the same -schema with an `anthropic/claude-...` model name: +SkillHone starts and stops a loopback-only LiteLLM proxy automatically. The +user does not need to run LiteLLM or supply an Anthropic-compatible endpoint. +Anthropic uses the same schema with an `anthropic/claude-...` model name: ```jsonc { @@ -118,8 +105,6 @@ schema with an `anthropic/claude-...` model name: } ``` -The user does not need to run LiteLLM or supply `ANTHROPIC_BASE_URL`. - Then write `~/.skillhone/settings.json` yourself, following the schema in [`skills/skillhone/references/configuration.md`](../../skills/skillhone/references/configuration.md). diff --git a/skills/skillhone/scripts/evaluation/template.py b/skills/skillhone/scripts/evaluation/template.py index 50f0908..bcf744f 100644 --- a/skills/skillhone/scripts/evaluation/template.py +++ b/skills/skillhone/scripts/evaluation/template.py @@ -88,11 +88,17 @@ def _redact_for_log(value): _CURRENT_DATASET_DIR: Path | None = None +def _executor_profile() -> dict: + """Return the executor profile, falling back to the improver profile.""" + if not _cfg: + return {} + return _cfg.get("executor") or _cfg.get("improver") or {} + + def _get_api_base() -> str: - if _cfg and "executor" in _cfg: - v = _cfg["executor"].get("api_base") - if v: - return v + v = _executor_profile().get("api_base") + if v: + return v v = os.environ.get("EXECUTOR_API_BASE", "") if v: return v @@ -100,10 +106,9 @@ def _get_api_base() -> str: def _get_model() -> str: - if _cfg and "executor" in _cfg: - v = _cfg["executor"].get("model") - if v: - return v + v = _executor_profile().get("model") + if v: + return v v = os.environ.get("EXECUTOR_API_MODELS", "") if v: return v @@ -112,10 +117,9 @@ def _get_model() -> str: def _get_model_alias() -> str: """SDK model alias (e.g. 'haiku') that maps to the actual model via env.""" - if _cfg and "executor" in _cfg: - v = _cfg["executor"].get("sdk_model_alias") - if v: - return v + v = _executor_profile().get("sdk_model_alias") + if v: + return v return os.environ.get("ANTHROPIC_MODEL", "haiku") @@ -135,8 +139,9 @@ def _get_workers() -> int: env_v = os.environ.get("EXECUTOR_WORKERS") or os.environ.get("EXECUTOR_API_WORKERS") if env_v: return int(env_v) - if _cfg and "executor" in _cfg: - return _cfg["executor"].get("workers", 16) + profile = _executor_profile() + if profile: + return profile.get("workers", 16) return 16 @@ -146,8 +151,9 @@ def _get_timeout() -> int: env_v = os.environ.get("EXECUTOR_TIMEOUT") if env_v: return int(env_v) - if _cfg and "executor" in _cfg: - return _cfg["executor"].get("timeout", 1800) + profile = _executor_profile() + if profile: + return profile.get("timeout", 1800) return 1800 @@ -156,8 +162,9 @@ def _get_max_turns() -> int | None: env_v = os.environ.get("EXECUTOR_MAX_TURNS") if env_v: return int(env_v) if env_v.lower() not in ("", "none", "0") else None - if _cfg and "executor" in _cfg: - v = _cfg["executor"].get("max_turns") + profile = _executor_profile() + if profile: + v = profile.get("max_turns") return int(v) if v not in (None, 0) else None return None @@ -667,8 +674,9 @@ def _passthrough_env_names() -> tuple[str, ...]: solver agent. Read from settings.json `executor.passthrough_env` (a list of UPPER_CASE names). Empty by default — the harness has no opinion about which task-domain credentials a given skill needs.""" - if _cfg and "executor" in _cfg: - names = _cfg["executor"].get("passthrough_env") or [] + profile = _executor_profile() + if profile: + names = profile.get("passthrough_env") or [] if isinstance(names, list): return tuple(str(x) for x in names if x) return () @@ -678,14 +686,15 @@ def _build_agent_env() -> dict[str, str]: """Build environment variables for claude-agent-sdk from settings.json, plus any allowlisted task-domain credentials present in os.environ (controlled by `executor.passthrough_env`).""" - if _cfg and "executor" in _cfg and agent_env_for is not None: - env = agent_env_for(_cfg["executor"], _cfg.get("api_key", "")) - elif (_cfg and "executor" in _cfg - and ("/" in _cfg["executor"].get("model", "") - or _cfg["executor"].get("provider") == "litellm")): + profile = _executor_profile() + if profile and agent_env_for is not None: + env = agent_env_for(profile, _cfg.get("api_key", "") if _cfg else "") + elif (profile + and ("/" in profile.get("model", "") + or profile.get("provider") == "litellm")): raise RuntimeError("LiteLLM adapter not found; reinstall the SkillHone skill bundle") - elif _cfg and "executor" in _cfg and "env" in _cfg["executor"]: - env = {k: str(v) for k, v in _cfg["executor"]["env"].items()} + elif profile and "env" in profile: + env = {k: str(v) for k, v in profile["env"].items()} else: env = { "ANTHROPIC_BASE_URL": _get_api_base(), diff --git a/skills/skillhone/scripts/synth.py b/skills/skillhone/scripts/synth.py index de0d32b..099d126 100644 --- a/skills/skillhone/scripts/synth.py +++ b/skills/skillhone/scripts/synth.py @@ -44,6 +44,11 @@ from core.litellm_proxy import agent_env_for # noqa: E402 +def _model_profile(settings: dict, role: str) -> dict: + """Return a role profile, falling back to the improver profile.""" + return settings.get(role) or settings.get("improver", {}) + + def _make_run_id(repo_name: str) -> str: import datetime ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") @@ -553,16 +558,16 @@ def main() -> int: f"factually answerable does not override these gates.\n" ) - improver_cfg = settings.get("improver", {}) - model = improver_cfg.get("sdk_model_alias", "opus") - improver_env = agent_env_for(improver_cfg, settings.get("api_key", "")) - disallowed_tools = improver_cfg.get("disallowed_tools", ["WebSearch"]) + synthesis_cfg = _model_profile(settings, "synthesis") + model = synthesis_cfg.get("sdk_model_alias", "opus") + improver_env = agent_env_for(synthesis_cfg, settings.get("api_key", "")) + disallowed_tools = synthesis_cfg.get("disallowed_tools", ["WebSearch"]) # Allowlist of env-var names forwarded from the parent shell into the - # synth agent. Read from `improver.passthrough_env` (a list of UPPER_CASE + # synth agent. Read from `synthesis.passthrough_env` (a list of UPPER_CASE # names). Empty by default — task-domain credentials are opt-in per # deployment. passthrough_env = tuple( - str(x) for x in (improver_cfg.get("passthrough_env") or []) if x + str(x) for x in (synthesis_cfg.get("passthrough_env") or []) if x ) target_max = args.target_pass_rate_max diff --git a/tests/test_model_profiles.py b/tests/test_model_profiles.py new file mode 100644 index 0000000..9d0c2bb --- /dev/null +++ b/tests/test_model_profiles.py @@ -0,0 +1,46 @@ +import sys +from pathlib import Path + + +SCRIPTS = Path(__file__).parents[1] / "skills" / "skillhone" / "scripts" +sys.path.insert(0, str(SCRIPTS)) + +from evaluation import template +import synth + + +def test_executor_profile_falls_back_to_improver(monkeypatch): + improver = { + "model": "deepseek/deepseek-chat", + "api_key": "role-key", + "sdk_model_alias": "opus", + "workers": 3, + } + monkeypatch.setattr(template, "_cfg", {"improver": improver}) + + assert template._executor_profile() is improver + assert template._get_model() == "deepseek/deepseek-chat" + assert template._get_model_alias() == "opus" + assert template._get_workers() == 3 + + +def test_explicit_executor_profile_wins(monkeypatch): + executor = {"model": "openai/gpt-5-mini"} + monkeypatch.setattr(template, "_cfg", { + "improver": {"model": "deepseek/deepseek-chat"}, + "executor": executor, + }) + + assert template._executor_profile() is executor + assert template._get_model() == "openai/gpt-5-mini" + + +def test_synthesis_profile_is_independent_and_optional(): + improver = {"model": "deepseek/deepseek-chat"} + synthesis = {"model": "gemini/gemini-2.5-flash"} + + assert synth._model_profile({"improver": improver}, "synthesis") is improver + assert synth._model_profile({ + "improver": improver, + "synthesis": synthesis, + }, "synthesis") is synthesis