feat: local runner + stealth scraping backends (Camoufox, Scweet)

CLAUDE.md

@@ -21,7 +21,8 @@ devkit is a Claude Code plugin: deterministic YAML workflow engine, thin-dispatc
 | `src/mcp/server.go` | MCP server implementation | Tool dispatching |
 | `src/mcp/principles.go` | Principle injection | How `_principles.yml` reaches workflows |
 | `src/runners/runner.go` | Model tier interface | The `smart`/`general`/`fast` contract |
-| `src/runners/{claude,codex,gemini}.go` | Tier implementations | How each external CLI is called |
+| `src/runners/{claude,codex,gemini}.go` | Cloud tier implementations | How each external CLI is called |
+| `src/runners/local.go` | Local runner (Ollama/llama-server/vLLM) | OpenAI-compatible HTTP client; opt-in via `DEVKIT_LOCAL_ENABLED=1` |
 | `src/lib/state.go` + `state_json.go` | Workflow state persistence | Running-state schema, step outputs |
 | `src/lib/state_lock_{unix,windows}.go` | Cross-platform file locking | OS-specific state lock |
 | `src/lib/git.go` | Git helpers | Diff collection, branch checks |

README.md

@@ -288,7 +288,8 @@ Enforcement:
   └── Stop hook — blocks session end during active workflows
 
 Terminal usage (devkit workflow <name> "<description>"):
-  └── Subprocess runners for Codex/Gemini CLI usage
+  ├── Subprocess runners for Codex/Gemini CLI
+  └── In-process HTTP runner for Ollama/llama-server/vLLM (opt-in via DEVKIT_LOCAL_ENABLED=1)
 ```
 
 ---
@@ -306,7 +307,7 @@ devkit/
 ├── src/               # Go engine + MCP server
 │   ├── mcp/           # MCP server (tools, principles loader, session management)
 │   ├── engine/        # YAML workflow engine (parser, executor, tests)
-│   ├── runners/       # Codex, Gemini interfaces (terminal fallback)
+│   ├── runners/       # Claude/Codex/Gemini CLI interfaces + local HTTP runner (Ollama-compat, opt-in)
 │   ├── lib/           # DB, git, metrics, session state, reporting
 │   └── cmd/           # CLI entry points (including `devkit mcp`)
 ├── bin/               # devkit wrapper (committed) + downloaded engine binaries (gitignored)

ROADMAP.md

@@ -27,6 +27,8 @@
 - **Cross-domain dirty-bit enforcement** — Blocks completion without test evidence per domain
 - **Language-universal hooks** — Consolidated language-specific hooks into `lang-review.sh` with Go, TypeScript, Rust, Python, and Shell support
 - **Hook consolidation** — Merged 14 hooks into 10, reduced per-edit shell processes from 7 to 4
+- **Local runner** — OpenAI-compatible HTTP runner for Ollama/llama-server/vLLM alongside Claude/Codex/Gemini CLI runners. Opt-in via `DEVKIT_LOCAL_ENABLED=1`; defaults to `qwen3:32b` at `localhost:11434/v1`. Intended for `fast`-tier dispatch; cloud runners stay default for `smart`/`general` given local tool-call reliability gap.
+- **Stealth scraping backends** — `scrape` skill adds Camoufox (patched Firefox with leak-fixed JA3/TLS fingerprints) between Playwright and Firecrawl with auto-fallback on Cloudflare/DataDome/Incapsula challenge-page signatures, plus Scweet as host-routed backend for `x.com`/`twitter.com` URLs when `SCWEET_AUTH_TOKEN` is set.
 
 ## Retired
 

skills/scrape/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: scrape
-description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Playwright, Firecrawl, or WebFetch.
+description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Playwright, Camoufox (TLS-spoofing Firefox), Scweet (X/Twitter), Firecrawl, or WebFetch.
 ---
 
 # Web Scrape to Markdown
@@ -26,8 +26,10 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe
 
 1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs.
 2. **Playwright** — use if `npx playwright --version` succeeds (optional dep, install with `npx playwright install chromium`). Best for JS-heavy SPAs and sites that block headless scrapers. Free and local — no API keys.
-3. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Paid API. Good for anti-bot bypass when Playwright isn't enough.
-4. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean).
+3. **Camoufox** — patched Firefox with leak-fixed JA3/TLS fingerprints. Use if `python -m camoufox version` succeeds (install: `pip install -U camoufox[geoip] && python -m camoufox fetch`). Free and local. Use when Playwright returns a challenge page (Cloudflare "Just a moment...", DataDome, PerimeterX). Auto-triggered when Playwright response body matches challenge signatures (`cf-chl-bypass`, `__cf_chl_`, `_Incapsula_Resource`).
+4. **Scweet (X/Twitter only)** — specialized scraper for `x.com` / `twitter.com` URLs. Use if `SCWEET_COOKIES_PATH` is set (points at a JSON cookie file exported from a logged-in browser). Auto-routed when host is `x.com` or `twitter.com` — other backends don't work reliably on X post-API lockdown.
+5. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Paid API. Last-resort anti-bot bypass when stealth browsers still get blocked.
+6. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean).
 
 ## Execution
 
@@ -55,7 +57,10 @@ Check availability: `npx playwright --version`. If not installed, tell the user
 Never build scripts via inline `-e "..."` strings. Write the script to a file that reads the URL from `process.argv[2]`, then invoke it:
 
 ```bash
-cat > /tmp/devkit-scrape.mjs <<'EOF'
+DRIVER=$(mktemp -t devkit-scrape.XXXXXX.mjs)
+trap 'rm -f "$DRIVER"' EXIT
+
+cat > "$DRIVER" <<'EOF'
 import { chromium } from 'playwright';
 
 const url = process.argv[2];
@@ -64,25 +69,110 @@ if (!url) { console.error('usage: node devkit-scrape.mjs <url>'); process.exit(2
 const browser = await chromium.launch();
 try {
   const page = await browser.newPage();
-  await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });
+  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
   const title = await page.title();
   const html = await page.content();
   process.stdout.write(JSON.stringify({ title, html }));
 } catch (err) {
   console.error('playwright failed:', err.message);
   process.exit(1);
 } finally {
-  try { await browser.close(); } catch (e) { /* suppress close errors */ }
+  try { await browser.close(); } catch (e) { console.error('playwright close error (browser may have crashed mid-scrape):', e.message); }
 }
 EOF
 
-node /tmp/devkit-scrape.mjs "$URL"
+node "$DRIVER" "$URL"
 ```
 
 Pass the URL as a shell variable (`URL=https://example.com`), never inline into the command string. The `.mjs` reads it from `process.argv[2]` so quotes, backticks, and `$()` in the URL can't escape.
 
 Then convert HTML → Markdown. Prefer a local converter if available (pandoc, turndown). If none available, strip script/style/nav/footer tags and extract text from article/main/body.
 
+**Camoufox (when Playwright returns a challenge page or `--backend camoufox`):**
+
+Check availability: `python -m camoufox version`. If missing, print once per session: `Camoufox not installed. Install with: pip install -U camoufox[geoip] && python -m camoufox fetch`, then fall through.
+
+Write the driver to a file (never inline `-c "..."` — same reasoning as Playwright). Pass the URL as an env var, never interpolated into the command string. Use `mktemp` rather than a hardcoded `/tmp/` path (portability + predictable-path attack surface):
+
+```bash
+DRIVER=$(mktemp -t devkit-camoufox.XXXXXX.py)
+trap 'rm -f "$DRIVER"' EXIT
+
+cat > "$DRIVER" <<'EOF'
+import os, json, sys, traceback
+from camoufox.sync_api import Camoufox
+
+url = os.environ["DEVKIT_SCRAPE_URL"]
+stage = "init"
+try:
+    with Camoufox(headless=True, humanize=True) as browser:
+        page = browser.new_page()
+        stage = "goto"
+        page.goto(url, wait_until="domcontentloaded", timeout=30000)
+        stage = "title"
+        title = page.title()
+        stage = "content"
+        html = page.content()
+        print(json.dumps({"title": title, "html": html}))
+except Exception as err:
+    print(json.dumps({"error": str(err), "stage": stage, "traceback": traceback.format_exc()[-2000:]}))
+    sys.exit(1)
+EOF
+
+DEVKIT_SCRAPE_URL="$URL" python "$DRIVER"
+```
+
+Then convert HTML → Markdown (same converter as Playwright). If the JSON has an `error` field, treat the backend as failed and fall through.
+
+**Challenge-page detection** (triggers auto-fallback Playwright → Camoufox): treat the Playwright result as blocked and retry with Camoufox when ANY of the following holds:
+
+- Positive anti-bot signature present: `cf-chl-bypass`, `__cf_chl_`, `<title>Just a moment...</title>`, `_Incapsula_Resource`, `dd_cookie_test_`, `datadome`, `px-captcha`, `perimeterx` (case-insensitive match against the HTML)
+- Body text under 200 chars AND no `<article>`/`<main>`/`<section>`/`<h1>` tag (skeleton/challenge shell, not a real small page)
+
+Narrower than a size heuristic alone — valid small pages (404s, hand-rolled minimal HTML, API docs) keep their original backend instead of burning a Camoufox cycle.
+
+**Scweet (X/Twitter — host is `x.com` or `twitter.com`):**
+
+Check availability: `python -c "import Scweet"`. If missing: `pip install Scweet`. Requires `SCWEET_COOKIES_PATH` pointing at a JSON cookies file exported from a logged-in browser session (devkit convention: export with a cookie extension, store outside the repo, set `SCWEET_COOKIES_PATH=~/.config/devkit/x-cookies.json`). Do NOT hardcode auth tokens in env.
+
+Scweet's Python API varies by version — the `scrape_tweets` / `scrape_profile` / `scrape_tweet` entry points differ across 2.x releases. Inspect the installed version with `python -c "import Scweet; print(Scweet.__version__)"` before wiring. Reference template (adapt to your installed Scweet version):
+
+```bash
+DRIVER=$(mktemp -t devkit-scweet.XXXXXX.py)
+trap 'rm -f "$DRIVER"' EXIT
+
+cat > "$DRIVER" <<'EOF'
+import os, json, sys, traceback, re
+from urllib.parse import urlparse
+
+url = os.environ["DEVKIT_SCRAPE_URL"]
+cookies_path = os.environ["SCWEET_COOKIES_PATH"]
+stage = "init"
+try:
+    # Scweet 2.x: choose entry point by URL shape
+    path = urlparse(url).path
+    if re.match(r"^/[^/]+/status/\d+", path):
+        stage = "import_tweet"
+        from Scweet.user import get_tweet
+        data = get_tweet(tweet_url=url, cookies_path=cookies_path, headless=True)
+    elif re.match(r"^/[^/]+/?$", path):
+        stage = "import_profile"
+        from Scweet.user import get_user_information
+        handle = path.strip("/")
+        data = get_user_information(handles=[handle], cookies_path=cookies_path, headless=True)
+    else:
+        raise ValueError(f"unsupported x.com path: {path}")
+    print(json.dumps({"data": data}, default=str))
+except Exception as err:
+    print(json.dumps({"error": str(err), "stage": stage, "traceback": traceback.format_exc()[-2000:]}))
+    sys.exit(1)
+EOF
+
+DEVKIT_SCRAPE_URL="$URL" python "$DRIVER"
+```
+
+Output is structured tweet/profile data — render to Markdown (username + timestamp + text + engagement stats for tweets; bio + follower counts for profiles). If the JSON has an `error` field, fall back to Jina.
+
 **Firecrawl (if FIRECRAWL_API_KEY is set and --backend firecrawl):**
 ```
 Use Bash to call (use jq to safely construct JSON — never interpolate URLs directly):
@@ -110,7 +200,7 @@ When given multiple URLs, scrape them in parallel:
 
 ### Error Handling
 
-**Fallback chain (single linear order):** Jina Reader → Playwright → Firecrawl → WebFetch. Each backend is tried in order. Move to the next only on error, empty content, or missing dependency.
+**Fallback chain (single linear order):** Jina Reader → Playwright → Camoufox → Firecrawl → WebFetch. Each backend is tried in order. Move to the next only on error, empty content, challenge-page detection, or missing dependency. **Exception:** URLs on `x.com` / `twitter.com` route directly to Scweet (skip the chain) when `SCWEET_COOKIES_PATH` is set; if Scweet is unavailable or fails, fall back to Jina.
 
 - **Report which backend actually ran** for each URL, and (if fallbacks happened) why the earlier ones failed. Never silently substitute content — the user must know a paywall/cookie-wall/SPA-skeleton from one backend wasn't the real page fetched by another.
 - If a URL is unreachable by every backend, report the error and continue with remaining URLs.

src/runners/local.go

@@ -0,0 +1,192 @@
+package runners
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strconv"
+	"time"
+)
+
+// LocalRunner dispatches to an OpenAI-compatible local endpoint (Ollama, llama-server, vLLM).
+// Gated behind DEVKIT_LOCAL_ENABLED=1 so it's opt-in. Tier assignment is the caller's
+// responsibility (see DetectRunners in runner.go) — this runner does not enforce tiering
+// itself; it's intended for fast-tier dispatch because local models have higher tool-call
+// error rates than cloud tiers.
+//
+// Config via env:
+//
+//	DEVKIT_LOCAL_ENABLED   — "1" to enable (default: disabled)
+//	DEVKIT_LOCAL_ENDPOINT  — base URL (default: http://localhost:11434/v1)
+//	DEVKIT_LOCAL_MODEL     — model name (default: qwen3:32b)
+//	DEVKIT_LOCAL_API_KEY   — optional bearer token
+//	DEVKIT_LOCAL_TIMEOUT   — per-request timeout in seconds (default: 600)
+//	DEVKIT_LOCAL_DEBUG     — "1" to log probe failures to stderr
+type LocalRunner struct{}
+
+type localRole string
+
+const (
+	roleSystem    localRole = "system"
+	roleUser      localRole = "user"
+	roleAssistant localRole = "assistant"
+)
+
+type localChatRequest struct {
+	Model    string         `json:"model"`
+	Messages []localMessage `json:"messages"`
+}
+
+type localMessage struct {
+	Role    localRole `json:"role"`
+	Content string    `json:"content"`
+}
+
+type localChatResponse struct {
+	ID      string `json:"id"`
+	Choices []struct {
+		Message      localMessage `json:"message"`
+		FinishReason string       `json:"finish_reason"`
+	} `json:"choices"`
+	Usage struct {
+		PromptTokens     int `json:"prompt_tokens"`
+		CompletionTokens int `json:"completion_tokens"`
+	} `json:"usage"`
+}
+
+func (r *LocalRunner) Name() string { return "local" }
+
+func (r *LocalRunner) Available() bool {
+	if os.Getenv("DEVKIT_LOCAL_ENABLED") != "1" {
+		return false
+	}
+	endpoint := localEndpoint()
+	req, err := http.NewRequest(http.MethodGet, endpoint+"/models", nil)
+	if err != nil {
+		localDebugf("local runner probe: building request for %s: %v", endpoint, err)
+		return false
+	}
+	if key := os.Getenv("DEVKIT_LOCAL_API_KEY"); key != "" {
+		req.Header.Set("Authorization", "Bearer "+key)
+	}
+	client := &http.Client{Timeout: 3 * time.Second}
+	resp, err := client.Do(req)
+	if err != nil {
+		localDebugf("local runner probe: %s unreachable: %v", endpoint, err)
+		return false
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode >= 400 {
+		localDebugf("local runner probe: %s returned HTTP %d (check DEVKIT_LOCAL_API_KEY / endpoint path)", endpoint, resp.StatusCode)
+		return false
+	}
+	return true
+}
+
+func (r *LocalRunner) Run(ctx context.Context, prompt string, opts RunOpts) (RunResult, error) {
+	messages := []localMessage{}
+	if opts.AppendSystemPrompt != "" {
+		messages = append(messages, localMessage{Role: roleSystem, Content: opts.AppendSystemPrompt})
+	}
+	if opts.AppendSystemPromptFile != "" {
+		data, err := os.ReadFile(opts.AppendSystemPromptFile)
+		if err != nil {
+			return RunResult{ExitCode: 1}, fmt.Errorf("reading system prompt file %q: %w", opts.AppendSystemPromptFile, err)
+		}
+		messages = append(messages, localMessage{Role: roleSystem, Content: string(data)})
+	}
+	messages = append(messages, localMessage{Role: roleUser, Content: prompt})
+
+	if len(messages) == 0 {
+		return RunResult{ExitCode: 1}, fmt.Errorf("local runner: no messages to send")
+	}
+
+	body, err := json.Marshal(localChatRequest{
+		Model:    localModel(),
+		Messages: messages,
+	})
+	if err != nil {
+		return RunResult{ExitCode: 1}, fmt.Errorf("marshaling local request: %w", err)
+	}
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, localEndpoint()+"/chat/completions", bytes.NewReader(body))
+	if err != nil {
+		return RunResult{ExitCode: 1}, fmt.Errorf("building local request: %w", err)
+	}
+	req.Header.Set("Content-Type", "application/json")
+	if key := os.Getenv("DEVKIT_LOCAL_API_KEY"); key != "" {
+		req.Header.Set("Authorization", "Bearer "+key)
+	}
+
+	client := &http.Client{Timeout: localTimeout()}
+	resp, err := client.Do(req)
+	if err != nil {
+		return RunResult{ExitCode: 1}, fmt.Errorf("local endpoint request failed: %w", err)
+	}
+	defer resp.Body.Close()
+
+	raw, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return RunResult{ExitCode: 1}, fmt.Errorf("reading local response: %w", err)
+	}
+
+	if resp.StatusCode >= 400 {
+		return RunResult{Output: string(raw), ExitCode: 1},
+			fmt.Errorf("local endpoint returned HTTP %d: %s", resp.StatusCode, TruncStr(string(raw), 200))
+	}
+
+	var parsed localChatResponse
+	if err := json.Unmarshal(raw, &parsed); err != nil {
+		return RunResult{Output: string(raw), ExitCode: 1},
+			fmt.Errorf("local endpoint returned non-JSON (%w): %s", err, TruncStr(string(raw), 200))
+	}
+
+	var output string
+	if len(parsed.Choices) > 0 {
+		output = parsed.Choices[0].Message.Content
+	}
+
+	return RunResult{
+		Output:    output,
+		SessionID: parsed.ID,
+		TokensIn:  parsed.Usage.PromptTokens,
+		TokensOut: parsed.Usage.CompletionTokens,
+		CostUSD:   0,
+		ExitCode:  0,
+	}, nil
+}
+
+func envDefault(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}
+
+func localEndpoint() string { return envDefault("DEVKIT_LOCAL_ENDPOINT", "http://localhost:11434/v1") }
+func localModel() string    { return envDefault("DEVKIT_LOCAL_MODEL", "qwen3:32b") }
+
+func localTimeout() time.Duration {
+	const fallback = 10 * time.Minute
+	raw := os.Getenv("DEVKIT_LOCAL_TIMEOUT")
+	if raw == "" {
+		return fallback
+	}
+	secs, err := strconv.Atoi(raw)
+	if err != nil || secs <= 0 {
+		localDebugf("local runner: ignoring DEVKIT_LOCAL_TIMEOUT=%q (want positive integer seconds)", raw)
+		return fallback
+	}
+	return time.Duration(secs) * time.Second
+}
+
+func localDebugf(format string, args ...any) {
+	if os.Getenv("DEVKIT_LOCAL_DEBUG") != "1" {
+		return
+	}
+	fmt.Fprintf(os.Stderr, format+"\n", args...)
+}