feat: local runner + stealth scraping backends (Camoufox, Scweet) - #90
Merged
Conversation
- src/runners/local.go: OpenAI-compatible runner for Ollama/llama-server/vLLM, opt-in via DEVKIT_LOCAL_ENABLED=1, defaults to qwen3:32b at localhost:11434. Intended for fast-tier dispatch only — local models have higher tool-call error rates that corrupt enforce:hard workflow state. - skills/scrape: add Camoufox (TLS-spoofing Firefox) between Playwright and Firecrawl with challenge-page auto-fallback (Cloudflare/DataDome/Incapsula signatures), and Scweet as a host-routed backend for x.com/twitter.com.
…lth scrape feature
local.go: - Available(): treat HTTP >=400 as unavailable (was <500); log probe failures via DEVKIT_LOCAL_DEBUG=1 so auth/endpoint misconfig is diagnosable - Run(): HTTP error path now sets ExitCode=1 (was leaking raw HTTP status); JSON unmarshal error wraps the underlying decode error with %w - Add localRole string type + const roleSystem/roleUser/roleAssistant — makes illegal roles unrepresentable at compile time - Add DEVKIT_LOCAL_TIMEOUT env knob (default 600s) - Extract envDefault helper; drop dead Stream:false field local_test.go (new): - Table-driven coverage via httptest.NewServer for: Available() gating across env + HTTP status matrix, Run() HTTP statuses (200/400/401/500/503 + non-JSON + empty choices), context cancellation, conditional auth header, network unreachable, system-prompt-file both branches, endpoint/model defaults, DEVKIT_LOCAL_TIMEOUT parsing edge cases (9 test groups, 41 subtests) scrape SKILL.md: - Replace hardcoded /tmp/devkit-*.* paths with mktemp + trap cleanup (Windows portability + predictable-path symlink attack surface, per common.md temp-file rule) - Camoufox + Scweet drivers now wrap execution in try/except emitting structured JSON error with stage + truncated traceback — bash caller can distinguish blocked page from crashed process - Narrow challenge-page detection: require positive anti-bot signature OR body-text<200chars+no-structural-tag (was "under 2KB no article/main", which false-positived on valid minimal pages) - Fix challenge signatures: drop incorrect DDG_ (DuckDuckGo), add dd_cookie_test_/datadome (actual DataDome), add px-captcha/perimeterx - Scweet driver: route by URL path (status → get_tweet, profile → get_user_information) matching real Scweet 2.x API; swap ad-hoc SCWEET_AUTH_TOKEN+SCWEET_CT0 for SCWEET_COOKIES_PATH pointing at a JSON cookies file (standard Scweet input, no creds in env) - Playwright driver: networkidle → domcontentloaded (avoids hangs on persistent websockets); browser.close() error now logged to stderr instead of fully suppressed README.md: - Architecture diagram: split subprocess runners (Codex/Gemini) from in-process HTTP runner (local) — they're categorically different
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/runners/local.go) for Ollama/llama-server/vLLM. Opt-in viaDEVKIT_LOCAL_ENABLED=1; defaults toqwen3:32bathttp://localhost:11434/v1. Env overrides:DEVKIT_LOCAL_ENDPOINT,DEVKIT_LOCAL_MODEL,DEVKIT_LOCAL_API_KEY. Intended forfast-tier dispatch — cloud runners remain default forsmart/general.scrapeskill with Camoufox (patched Firefox, leak-fixed JA3/TLS fingerprints) between Playwright and Firecrawl with auto-fallback on Cloudflare/DataDome/Incapsula challenge-page signatures. Free and local.x.com/twitter.comURLs, gated behindSCWEET_AUTH_TOKEN+SCWEET_CT0.CLAUDE.md,README.md,ROADMAP.md,skills/scrape/SKILL.md.Context
Motivated by research into local inference stacks (RTX 5090 32GB + Apple Silicon) and anti-bot scraping. Local runner is deliberately opt-in and limited-scope: local 32B models have 15-30% tool-call failure rates on 4+ step chains vs <5% for Sonnet, which would corrupt
enforce: hardworkflow state if wired into smart/general tiers.Test plan
make check— go vet + gofmt cleango test ./...— 310 tests pass across 6 packagesDEVKIT_LOCAL_ENABLED=1with Ollama running andqwen3:32bpulled; confirmDetectRunners()returns the local runner and aRun()call completes/devkit:scrapeagainst a Cloudflare-protected URL and verify Playwright → Camoufox fallback triggers on challenge signature/devkit:scrape https://x.com/some/statuswithSCWEET_AUTH_TOKEN/SCWEET_CT0set and verify host routing skips the main chainSecurity
json.Marshal, no string interpolation@) applies to new backendsCommits
7ae1cf2feat: add local runner + stealth scraping backends17032dedocs: update CLAUDE.md, README.md, ROADMAP.md for local runner + stealth scrape feature