diff --git a/apps/docs/src/components/landing/Capabilities.astro b/apps/docs/src/components/landing/Capabilities.astro index 014b0e5..2986325 100644 --- a/apps/docs/src/components/landing/Capabilities.astro +++ b/apps/docs/src/components/landing/Capabilities.astro @@ -67,8 +67,8 @@ const capabilities: Capability[] = [ href: "/loop/gate-floor/", }, { - title: "Free web access", - body: "Local web_fetch + web_search — no API key, fully on-machine.", + title: "Keyless web research", + body: "Package metadata, docs, search, fetch, and local browser reads — no vendor API key.", href: "/integrations/web-tools/", }, ]; diff --git a/apps/docs/src/content/docs/agent/model-agent.mdx b/apps/docs/src/content/docs/agent/model-agent.mdx index 373652b..361f925 100644 --- a/apps/docs/src/content/docs/agent/model-agent.mdx +++ b/apps/docs/src/content/docs/agent/model-agent.mdx @@ -24,7 +24,7 @@ One approved task can involve many agent cycles until the gate passes or tsforge | Navigation | `search`, `symbol_search`, `find_references`, `type_at`, `diagnostics`, `rename_symbol`, `move_file`, `organize_imports` | existing-code repos | | Git context | `git_context` | existing-code repos (read-only: diff/log/blame/show to scope a change); `TSFORGE_NO_GIT_TOOL=1` to withhold | | Web | `scaffold_web`, `scaffold_ui`, `scaffold_routes`, `add_dependency` | web builds | -| Web access | `web_fetch`, `web_search` | when `TSFORGE_WEB=1` (free, local — no API key) | +| Web research | `package_info`, `package_docs`, `web_fetch`, `web_search`, `web_browse` | when `TSFORGE_WEB=1` (no required API keys or paid browser/search service) | | Control | `yield_status` | end turn with a summary | On greenfield specs, navigation tools are often withheld so the model focuses on creating files instead of exploring an empty tree. See [TypeScript language server](/lsp/typescript-server/). diff --git a/apps/docs/src/content/docs/integrations/web-tools.mdx b/apps/docs/src/content/docs/integrations/web-tools.mdx index 945dd55..be586c3 100644 --- a/apps/docs/src/content/docs/integrations/web-tools.mdx +++ b/apps/docs/src/content/docs/integrations/web-tools.mdx @@ -1,9 +1,9 @@ --- -title: Web access (free, local) -description: "Opt-in web_fetch + web_search tools — no API key, no paid service. Fully local extraction, DuckDuckGo by default, SearXNG for full privacy." +title: Web research (no API keys) +description: "Opt-in web_fetch, web_search, package_info, package_docs, and web_browse tools — no paid search/browser API, no required service key." --- -Set `TSFORGE_WEB=1` to give the agent two read-only web tools. They're **free and fully local** — no API key, no paid search service, nothing leaves your machine except the fetch itself. Off by default, so a run without the flag has no network reach beyond your model endpoint. +Set `TSFORGE_WEB=1` to give the agent read-only internet research tools. They're built for **no required API keys and no paid vendor coupling**: npm metadata comes from the configured registry, search defaults to DuckDuckGo's keyless HTML endpoint, pages are extracted locally, and browser rendering uses local Playwright/Chromium when available. Off by default, so a run without the flag has no network reach beyond your model endpoint. ```bash TSFORGE_WEB=1 tsforge "update the deprecated API call — check the library's current docs" @@ -11,20 +11,33 @@ TSFORGE_WEB=1 tsforge "update the deprecated API call — check the library's cu ## The tools +- **`package_info`** — read current npm package metadata from the configured registry: latest dist-tag, versions, deprecation status, peer deps, homepage, repository, and dependency names. +- **`package_docs`** — read package docs without a docs API: local `node_modules` README/package metadata/types first, then the npm registry README when needed. - **`web_fetch`** — retrieve a URL and return readable markdown. The page is fetched and the content extracted **on-machine** (no third-party reader API), so you get clean text without shipping the URL to a service. -- **`web_search`** — discover sources for a query. Defaults to DuckDuckGo's keyless HTML endpoint; point it at a self-hosted [SearXNG](https://docs.searxng.org/) instance for full privacy and control: +- **`web_browse`** — open a public URL in local headless Chromium via Playwright and return rendered visible text, final URL, title, and links. Use it for JavaScript-rendered docs that `web_fetch` cannot see. No hosted browser service. +- **`web_search`** — discover sources for a query. It returns public result URLs only, supports `recency` (`day`, `month`, `year`), `domains` for official-site scoping, and `maxResults` up to 20. Defaults to DuckDuckGo's keyless HTML endpoint. + +SearXNG is **not bundled or started by tsforge**. If you run your own SearXNG service (Docker, Compose, or any deployment), point `TSFORGE_SEARXNG_URL` at it for full privacy/control: ```bash TSFORGE_WEB=1 TSFORGE_SEARXNG_URL=http://localhost:8888 tsforge ``` +For current TypeScript/library work, ask the agent to search the official host first, then fetch the source it picks: + +```text +Check the current TanStack Query docs before changing this hook. Use domain-scoped web search if needed. +``` + ## Why opt-in The tools are read-only and offline-safe, but web access is still more reach than the agent has by default — so it's a deliberate flag, not an always-on capability. Under a policy mode that denies `network` (e.g. `ci`), the tools are unavailable even with the flag set. See [Permissions & policy](/guardrails/policy/). | Env var | Default | Effect | | --- | --- | --- | -| `TSFORGE_WEB` | off | enable `web_fetch` + `web_search` (`=1`) | -| `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a self-hosted SearXNG instance | +| `TSFORGE_WEB` | off | enable keyless research tools (`=1`) | +| `TSFORGE_NPM_REGISTRY` | npm registry | registry used by `package_info` / `package_docs` | +| `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a SearXNG instance you already run | +| `TSFORGE_WEB_SEARCH_BACKEND` | auto | `duckduckgo` or `searxng`; `searxng` fails closed if no SearXNG URL is set | → [Environment variables](/reference/flags/) · [MCP servers](/integrations/mcp/) · [Permissions & policy](/guardrails/policy/) diff --git a/apps/docs/src/content/docs/reference/flags.mdx b/apps/docs/src/content/docs/reference/flags.mdx index 8c769a3..8a01f09 100644 --- a/apps/docs/src/content/docs/reference/flags.mdx +++ b/apps/docs/src/content/docs/reference/flags.mdx @@ -16,7 +16,7 @@ description: Canonical list of every TSFORGE_* environment variable. | `TSFORGE_FORCE_TOOLS` | off | force tool_choice required (`=1`) | | `TSFORGE_SIMPLICITY` | off | shortest-solution guidance, scratch non-web (`=1`) | | `TSFORGE_TDD` | ON (`≠ "0"`) | test-first guidance + `test-sibling-required` is an error on changed logic files (`=0` to opt out) | -| `TSFORGE_WEB` | off | free, local web access — the `web_fetch` + `web_search` tools (`=1`) | +| `TSFORGE_WEB` | off | keyless web/package research tools (`=1`) | | `TSFORGE_CONTRACT` | off | experimental per-feature [contract negotiation](/loop/greenfield/) in greenfield builds (`=1`) | | `TSFORGE_NO_UPDATE_CHECK` | off | silence the startup "update available" check (`=1`) | | `TSFORGE_NO_GIT_TOOL` | off | withhold the `git_context` tool (`=1`) | @@ -33,12 +33,14 @@ Offered only when there is existing code to inspect (greenfield scratch builds h ## Web access -Opt-in, free, and fully local — no API key, no paid service. `TSFORGE_WEB=1` adds two read-only tools: `web_fetch` (retrieve a URL → readable markdown, extracted on-machine) and `web_search` (discover sources). Search defaults to DuckDuckGo's keyless HTML endpoint; point at a self-hosted [SearXNG](https://docs.searxng.org/) instance for full privacy/control. Full guide: [Web access](/integrations/web-tools/). +Opt-in, free, and no required service keys. `TSFORGE_WEB=1` adds read-only research tools: `package_info`, `package_docs`, `web_fetch`, `web_search`, and `web_browse`. Search defaults to DuckDuckGo's keyless HTML endpoint. SearXNG is not bundled; set `TSFORGE_SEARXNG_URL` only when you already run a SearXNG service. Full guide: [Web access](/integrations/web-tools/). | Variable | Default | Toggles | | --- | --- | --- | -| `TSFORGE_WEB` | off | enable `web_fetch` + `web_search` (`=1`) | -| `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a self-hosted SearXNG instance (e.g. `http://localhost:8888`) | +| `TSFORGE_WEB` | off | enable keyless web/package research tools (`=1`) | +| `TSFORGE_NPM_REGISTRY` | npm registry | registry used by `package_info` / `package_docs` | +| `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a SearXNG instance you already run (e.g. `http://localhost:8888`) | +| `TSFORGE_WEB_SEARCH_BACKEND` | auto | `duckduckgo` or `searxng`; `searxng` fails closed if no SearXNG URL is set | ## Opt-in gate oracles diff --git a/apps/docs/src/content/docs/reference/roadmap.mdx b/apps/docs/src/content/docs/reference/roadmap.mdx index 50f71be..58ca2b6 100644 --- a/apps/docs/src/content/docs/reference/roadmap.mdx +++ b/apps/docs/src/content/docs/reference/roadmap.mdx @@ -19,7 +19,7 @@ TypeScript coding harness for web projects — `packages/core` for the loop and **The agent surface** - **Functional review** (`tsforge review`, gate-aware) · **workspace map** (`tsforge map`) · **pre-edit scout** · **declarative recipes** (`tsforge run`) · **plan mode** · **cross-session memory**. - **Permissions & policy** — 6 modes, deny-first config rules, and a run **ledger** summarized by `tsforge trace`. See [Permissions & policy](/guardrails/policy/) · [Trace a run](/observability/trace/). -- **Free, local web access** — opt-in `web_fetch` + `web_search`, no API key. See [Web access](/integrations/web-tools/). +- **Keyless web research** — opt-in package metadata/docs, search, fetch, and local browser reads with no required API key. See [Web access](/integrations/web-tools/). - **Web scaffolding** — Vite + React + shadcn + TanStack, ending in a real browser smoke test. See [Web scaffolding](/scaffold/web/). **Models & observability** diff --git a/packages/core/src/agent/agent.constants.ts b/packages/core/src/agent/agent.constants.ts index bdacc1a..9a96fca 100644 --- a/packages/core/src/agent/agent.constants.ts +++ b/packages/core/src/agent/agent.constants.ts @@ -23,8 +23,11 @@ export const TOOL_NAME = { scaffoldRoutes: "scaffold_routes", scaffoldWeb: "scaffold_web", addDependency: "add_dependency", + packageInfo: "package_info", + packageDocs: "package_docs", webFetch: "web_fetch", webSearch: "web_search", + webBrowse: "web_browse", yieldStatus: "yield_status", } as const; @@ -41,11 +44,14 @@ export const READ_ONLY_TOOL_NAMES: ReadonlySet = new Set([ // git_context only inspects history/diffs — no workspace mutation — so it is a // plan-mode tool too (scope a review/fix while planning, before any edit). TOOL_NAME.gitContext, + TOOL_NAME.packageInfo, + TOOL_NAME.packageDocs, // Web tools are read-only (no workspace mutation), so they're usable in plan // mode too — research while planning. Network egress here is structured and // opt-in (TSFORGE_WEB), unlike the raw `run` curl path plan mode blocks. TOOL_NAME.webFetch, TOOL_NAME.webSearch, + TOOL_NAME.webBrowse, ]); /** The model's own decision to start a from-scratch WEB app: scaffolds the stack @@ -254,17 +260,116 @@ export const WEB_SEARCH_TOOL = { function: { name: TOOL_NAME.webSearch, description: - "Search the web and get back ranked result titles, URLs, and snippets. Use it to DISCOVER sources when you don't already have a URL, then `web_fetch` the most relevant one. Free and keyless — DuckDuckGo by default, or a self-hosted SearXNG instance via the TSFORGE_SEARXNG_URL env var.", + "Search the web and get back ranked public result titles, URLs, and snippets. Use it to DISCOVER current sources when you don't already have a URL, then `web_fetch` the most relevant one. Supports `recency` for fresh docs/news, `domains` for official-site scoping, and `maxResults` for broader source discovery. Free and keyless — DuckDuckGo by default, or a user-run SearXNG instance via TSFORGE_SEARXNG_URL. Set TSFORGE_WEB_SEARCH_BACKEND=searxng to fail closed instead of falling back to DuckDuckGo.", parameters: { type: "object", properties: { query: { type: "string", description: "the search query" }, + recency: { + type: "string", + enum: ["day", "month", "year"], + description: + "optional freshness window for fast-moving topics and current docs", + }, + domains: { + type: "array", + items: { type: "string" }, + description: + "optional public hostnames to search within, e.g. ['typescriptlang.org', 'nodejs.org']", + }, + maxResults: { + type: "number", + description: + "optional result cap (default 8, maximum 20) when comparing multiple sources", + }, }, required: ["query"], }, }, }; +export const WEB_BROWSE_TOOL = { + type: "function", + function: { + name: TOOL_NAME.webBrowse, + description: + "Open a public URL in a local headless Chromium browser via Playwright and return rendered visible text, final URL, title, and links. Use it when docs/sites require JavaScript or when web_fetch misses content. No hosted browser service or API key.", + parameters: { + type: "object", + properties: { + url: { + type: "string", + description: "absolute http(s) URL to open in the local browser", + }, + waitMs: { + type: "number", + description: + "optional extra wait after DOMContentLoaded for client-rendered docs (default 750, max 10000)", + }, + maxChars: { + type: "number", + description: "optional cap on returned visible text (default 10000)", + }, + }, + required: ["url"], + }, + }, +}; + +export const PACKAGE_INFO_TOOL = { + type: "function", + function: { + name: TOOL_NAME.packageInfo, + description: + "Read current npm package metadata from the configured npm registry with no API key: latest dist-tag, versions, deprecation, peer deps, homepage, repository, and dependency names. Use before installing or coding against a package API.", + parameters: { + type: "object", + properties: { + package: { + type: "string", + description: + "one npm package name, optionally @versioned, e.g. 'zod', 'react@19', '@tanstack/react-query'", + }, + maxChars: { + type: "number", + description: "optional cap on returned characters (default 12000)", + }, + }, + required: ["package"], + }, + }, +}; + +export const PACKAGE_DOCS_TOOL = { + type: "function", + function: { + name: TOOL_NAME.packageDocs, + description: + "Read package documentation with no paid service: local node_modules README/package.json/types first, then npm registry README when needed. Use this for version-aware package docs before guessing APIs.", + parameters: { + type: "object", + properties: { + package: { + type: "string", + description: + "one npm package name, optionally @versioned, e.g. 'zod@4' or '@tanstack/react-query'", + }, + source: { + type: "string", + enum: ["auto", "local", "registry"], + description: + "auto prefers installed local docs, local refuses network, registry uses npm metadata", + }, + maxChars: { + type: "number", + description: "optional cap on returned characters (default 12000)", + }, + }, + required: ["package"], + }, + }, +}; + /** * Semantic + search tools backed by the in-process TypeScript LanguageService * (+ ripgrep). Read-only tools (find_references, type_at, symbol_search, diff --git a/packages/core/src/loop/prompt/prompt.ts b/packages/core/src/loop/prompt/prompt.ts index d6894db..8511d03 100644 --- a/packages/core/src/loop/prompt/prompt.ts +++ b/packages/core/src/loop/prompt/prompt.ts @@ -41,6 +41,21 @@ export function buildSystem(conventions: IConventions): string { * don't thread conventions; the dynamic path uses {@link buildSystem}. */ export const SYSTEM = buildSystem(DEFAULT_CONVENTIONS); +export function buildWebResearchGuidance(): string { + return [ + "WEB RESEARCH — keyless internet/package tools are enabled:", + " • Package version/install question? Use `package_info` for npm registry", + " dist-tags, versions, deprecations, peer deps, homepage, and repo.", + " • Package API/docs question? Use `package_docs` first; it reads installed", + " node_modules docs/types before falling back to the npm registry README.", + " • Unknown public source? Use `web_search`, preferring official hosts with", + " `domains` and `recency` (`day`/`month`/`year`) for fast-moving topics.", + " • Known static page? Use `web_fetch`. JS-rendered docs/site? Use", + " `web_browse`, which opens the URL in local Chromium via Playwright.", + " • Do not guess APIs from memory when the user asks for latest/current info.", + ].join("\n"); +} + /** Appended to SYSTEM for from-scratch, NON-web utility builds when the simplicity * flag is on. Pushes the model toward the shortest correct solution — the axis the * gate is blind to (it checks correctness, never concision). Carve-outs keep it @@ -104,6 +119,10 @@ export function buildSystemPrompt( const webish = stack !== undefined && isWebStack(stack); const blocks: string[] = [buildSystem(conventions)]; + if (flags.webTools()) { + blocks.push(buildWebResearchGuidance()); + } + // Simplicity: from-scratch, non-web only (an A/B-gated concision push). if (flags.simplicity() && !hasExistingCode && !webish) { blocks.push(buildScratchSimplicityGuidance(conventions)); @@ -128,8 +147,7 @@ export function buildSystemPrompt( export function buildChatSystem(conventions: IConventions): string { const naming = interfaceNamingPhrase(conventions); const namingPart = naming === null ? "" : `${naming}; `; - - return [ + const lines = [ "You are tsforge, an expert TypeScript coding assistant. You are launched inside a repository, but NOT every request is about that repository. The user talks to you; you help by answering, and by inspecting/changing code with your tools.", "Tools: `read` (inspect a file), `run` (execute any shell command — `ls`, `rg`, tests, `tsc`), `edit` (replace an exact, unique snippet), `create` (a new file).", "File paths are RELATIVE to the workspace root: use `tsconfig.json` or `src/app.ts` — never an absolute path, and never repeat the workspace folder in the path.", @@ -138,7 +156,15 @@ export function buildChatSystem(conventions: IConventions): string { "Be decisive, not exhaustive. When you do investigate, a few targeted reads beat reading everything — as soon as you can answer or act, STOP calling tools and reply.", "For a QUESTION about the repo, investigate briefly then give a concise, concrete answer (cite specific files/symbols; offer your top few recommendations, not a survey). For a CHANGE, make it with `edit`/`create`, verify by `run`ning the tests or `tsc`, then briefly state what you did.", `When you write code, use strict TypeScript: ${namingPart}\`===\`; no \`var\`; never the non-null \`!\` (guard index access: \`const x = arr[i]; if (x === undefined) {…}\`); no \`any\`/\`as\` (type parameters); explicit boolean conditions.`, - ].join("\n"); + ]; + + if (flags.webTools()) { + lines.push( + "Web tools are enabled: use `package_info` for latest npm metadata, `package_docs` for package APIs, `web_search` to discover current sources, `web_fetch` for static pages, and `web_browse` for JS-rendered pages. Prefer official sources; use `domains`, `recency`, and `maxResults` when searching." + ); + } + + return lines.join("\n"); } /** Default-conventions interactive prompt (back-compat constant). */ diff --git a/packages/core/src/loop/tools/execute-tool.ts b/packages/core/src/loop/tools/execute-tool.ts index dd9e7ec..20ad146 100644 --- a/packages/core/src/loop/tools/execute-tool.ts +++ b/packages/core/src/loop/tools/execute-tool.ts @@ -10,6 +10,8 @@ import { doScaffoldWeb } from "./scaffold-web"; import { doAddDependency } from "./add-dependency"; import { doWebFetch } from "./web-fetch"; import { doWebSearch } from "./web-search"; +import { doWebBrowse } from "./web-browse"; +import { doPackageInfo, doPackageDocs } from "./package-info"; import { reject, type IToolContext } from "./tool-context"; import { classifyAction, @@ -45,8 +47,11 @@ const HANDLERS: Record = { [TOOL_NAME.scaffoldRoutes]: doScaffoldRoutes, [TOOL_NAME.scaffoldWeb]: doScaffoldWeb, [TOOL_NAME.addDependency]: doAddDependency, + [TOOL_NAME.packageInfo]: doPackageInfo, + [TOOL_NAME.packageDocs]: doPackageDocs, [TOOL_NAME.webFetch]: doWebFetch, [TOOL_NAME.webSearch]: doWebSearch, + [TOOL_NAME.webBrowse]: doWebBrowse, // yield_status is intercepted by the Session BEFORE tool dispatch (it ends the // turn); this handler only fires if one slips through with other calls. [TOOL_NAME.yieldStatus]: () => diff --git a/packages/core/src/loop/tools/package-info.ts b/packages/core/src/loop/tools/package-info.ts new file mode 100644 index 0000000..14b63eb --- /dev/null +++ b/packages/core/src/loop/tools/package-info.ts @@ -0,0 +1,651 @@ +import { join, resolve, relative, isAbsolute, dirname } from "node:path"; +import { isRecord } from "../../lib/guards"; +import { reject, str, type IToolContext } from "./tool-context"; +import { parsePackageSpecs } from "./add-dependency"; + +const DEFAULT_REGISTRY = "https://registry.npmjs.org"; +const DEFAULT_MAX_CHARS = 12_000; +const MAX_ALLOWED_CHARS = 48_000; + +export interface IPackageFetchResponse { + ok: boolean; + status: number; + json(): Promise; +} + +export interface IPackageInfoDeps { + fetchFn: (url: string) => Promise; +} + +interface IPackageDocHit { + source: "local" | "registry"; + content: string; +} + +function registryRoot(): string { + const configured = + process.env.TSFORGE_NPM_REGISTRY?.trim() ?? + process.env.NPM_CONFIG_REGISTRY?.trim() ?? + process.env.npm_config_registry?.trim() ?? + ""; + + return (configured.length > 0 ? configured : DEFAULT_REGISTRY).replace( + /\/+$/u, + "" + ); +} + +function singlePackageSpec(raw: string): string | null { + const specs = parsePackageSpecs(raw); + + if (specs?.length !== 1) { + return null; + } + + return specs[0] ?? null; +} + +function firstNonEmpty(left: string, right: string): string { + return left.length > 0 ? left : right; +} + +export function packageNameFromSpec(raw: string): string | null { + const spec = singlePackageSpec(raw); + + if (spec === null) { + return null; + } + + if (spec.startsWith("@")) { + const slash = spec.indexOf("/"); + + if (slash === -1) { + return null; + } + + const versionAt = spec.indexOf("@", slash + 1); + + return versionAt === -1 ? spec : spec.slice(0, versionAt); + } + + const versionAt = spec.indexOf("@"); + + return versionAt === -1 ? spec : spec.slice(0, versionAt); +} + +function versionFromSpec(raw: string): string | null { + const spec = singlePackageSpec(raw); + + if (spec === null) { + return null; + } + + if (spec.startsWith("@")) { + const slash = spec.indexOf("/"); + + if (slash === -1) { + return null; + } + + const versionAt = spec.indexOf("@", slash + 1); + + return versionAt === -1 ? null : spec.slice(versionAt + 1); + } + + const versionAt = spec.indexOf("@"); + + return versionAt === -1 ? null : spec.slice(versionAt + 1); +} + +function registryUrl(packageName: string): string { + return `${registryRoot()}/${encodeURIComponent(packageName)}`; +} + +function stringProp(record: Record, key: string): string { + const value = record[key]; + + return typeof value === "string" ? value : ""; +} + +function recordKeys(value: unknown): string[] { + return isRecord(value) ? Object.keys(value).sort() : []; +} + +function numericParts(version: string): number[] { + const main = version.split("+")[0]?.split("-")[0] ?? ""; + + return main.split(".").map((part) => { + const n = Number.parseInt(part, 10); + + return Number.isFinite(n) ? n : 0; + }); +} + +function comparePrerelease(a: string, b: string): number { + // A release with no prerelease tag outranks any prerelease (1.0.0 > 1.0.0-rc). + const aPre = a.split("+")[0]?.split("-").slice(1).join("-") ?? ""; + const bPre = b.split("+")[0]?.split("-").slice(1).join("-") ?? ""; + + if (aPre === bPre) { + return 0; + } + + if (aPre.length === 0) { + return 1; + } + + if (bPre.length === 0) { + return -1; + } + + return aPre < bPre ? -1 : 1; +} + +/** Ascending semver order. npm's `versions` object has no guaranteed key order, + * and a plain lexical sort misorders releases (1.2.0 < 1.10.0 numerically but + * not as strings) — so the "recent" list and the no-dist-tag latest fallback + * must compare version components numerically. */ +function compareVersions(a: string, b: string): number { + const pa = numericParts(a); + const pb = numericParts(b); + const len = Math.max(pa.length, pb.length); + + for (let i = 0; i < len; i++) { + const diff = (pa[i] ?? 0) - (pb[i] ?? 0); + + if (diff !== 0) { + return diff; + } + } + + return comparePrerelease(a, b); +} + +function sortedVersionKeys(value: unknown): string[] { + return isRecord(value) ? Object.keys(value).sort(compareVersions) : []; +} + +function repositoryUrl(value: unknown): string { + if (typeof value === "string") { + return value; + } + + if (isRecord(value) && typeof value.url === "string") { + return value.url; + } + + return ""; +} + +/** Highest semver-sorted key that equals `prefix` or sits under it on a dot + * boundary (so prefix "1" matches "1.2.0" but never "10.0.0"). `keys` are + * already ascending, so the last match is the highest. */ +function highestWithPrefix( + keys: readonly string[], + prefix: string +): string | null { + if (prefix.length === 0) { + return null; + } + + const matches = keys.filter( + (key) => key === prefix || key.startsWith(`${prefix}.`) + ); + + return matches[matches.length - 1] ?? null; +} + +/** Resolve a requested spec — a dist-tag name (`latest`/`next`), an exact + * version, or a major/minor/range like `19` or `^19.0.0` — to a concrete + * version key. The npm manifest indexes by EXACT version, so without this a + * `react@19` request would miss versionRecord and report empty dependency + * lists. Best-effort (not full semver-range satisfaction): falls back to the + * raw spec when nothing matches. */ +function resolveRequested( + manifest: Record, + requested: string +): string { + const tags = manifest["dist-tags"]; + + if (isRecord(tags) && typeof tags[requested] === "string") { + return tags[requested]; + } + + const keys = sortedVersionKeys(manifest.versions); + + if (keys.includes(requested)) { + return requested; + } + + const cleaned = requested.replace(/^[\^~>=<\s]+/u, "").split("-")[0] ?? ""; + const segments = cleaned.split(".").filter((part) => /^[0-9]+$/u.test(part)); + + if (segments.length === 0) { + return requested; + } + + // `^` allows anything up to the next major → lock the major; `~` allows up to + // the next minor → lock major.minor; a bare partial (`19`, `19.1`) narrows + // from the most specific prefix outward to the major. + const major = segments[0] ?? ""; + const minor = segments.slice(0, 2).join("."); + const prefixes = requested.startsWith("^") + ? [major] + : requested.startsWith("~") + ? [minor, major] + : [segments.join("."), minor, major]; + + for (const prefix of prefixes) { + const hit = highestWithPrefix(keys, prefix); + + if (hit !== null) { + return hit; + } + } + + return requested; +} + +function selectedVersion( + manifest: Record, + requested: string | null +): string { + if (requested !== null && requested.length > 0) { + return resolveRequested(manifest, requested); + } + + const tags = manifest["dist-tags"]; + + if (isRecord(tags) && typeof tags.latest === "string") { + return tags.latest; + } + + const keys = sortedVersionKeys(manifest.versions); + + return keys[keys.length - 1] ?? ""; +} + +function versionRecord( + manifest: Record, + version: string +): Record | null { + const versions = manifest.versions; + + if (!isRecord(versions)) { + return null; + } + + const value = versions[version]; + + return isRecord(value) ? value : null; +} + +function formatTags(value: unknown): string { + if (!isRecord(value)) { + return "(none)"; + } + + const lines: string[] = []; + + for (const key of Object.keys(value).sort()) { + const tag = value[key]; + + if (typeof tag === "string") { + lines.push(`${key}: ${tag}`); + } + } + + return lines.length > 0 ? lines.join(", ") : "(none)"; +} + +function formatDependencyList(label: string, value: unknown): string { + const keys = recordKeys(value); + + return keys.length > 0 ? `${label}: ${keys.join(", ")}` : `${label}: (none)`; +} + +function formatPackageInfo( + manifest: Record, + requestedVersion: string | null +): string { + const packageName = stringProp(manifest, "name"); + const version = selectedVersion(manifest, requestedVersion); + const details = versionRecord(manifest, version); + const source = details ?? manifest; + const description = firstNonEmpty( + stringProp(source, "description"), + stringProp(manifest, "description") + ); + const homepage = firstNonEmpty( + stringProp(source, "homepage"), + stringProp(manifest, "homepage") + ); + const repo = firstNonEmpty( + repositoryUrl(source.repository), + repositoryUrl(manifest.repository) + ); + const deprecated = stringProp(source, "deprecated"); + const versions = sortedVersionKeys(manifest.versions); + const recent = versions.slice(Math.max(0, versions.length - 12)); + + return [ + `# ${packageName.length > 0 ? packageName : "(unknown package)"}`, + `registry: ${registryRoot()}`, + `selected: ${version.length > 0 ? version : "(unknown)"}`, + `dist-tags: ${formatTags(manifest["dist-tags"])}`, + description.length > 0 ? `description: ${description}` : "", + stringProp(source, "license").length > 0 + ? `license: ${stringProp(source, "license")}` + : "", + homepage.length > 0 ? `homepage: ${homepage}` : "", + repo.length > 0 ? `repository: ${repo}` : "", + deprecated.length > 0 ? `DEPRECATED: ${deprecated}` : "", + `versions: ${String(versions.length)} total${recent.length > 0 ? `; recent: ${recent.join(", ")}` : ""}`, + formatDependencyList("dependencies", source.dependencies), + formatDependencyList("peerDependencies", source.peerDependencies), + ] + .filter((line) => line.length > 0) + .join("\n"); +} + +function maxChars(args: Record): number { + const value = args.maxChars; + + if (typeof value === "number" && Number.isFinite(value) && value > 0) { + return Math.min(Math.floor(value), MAX_ALLOWED_CHARS); + } + + return DEFAULT_MAX_CHARS; +} + +function truncate(content: string, max: number): string { + const trimmed = content.trim(); + + if (trimmed.length <= max) { + return trimmed; + } + + return `${trimmed.slice(0, max)}\n\n...[truncated ${String(trimmed.length - max)} chars - raise maxChars to read more]`; +} + +async function fetchManifest( + packageName: string, + deps: IPackageInfoDeps +): Promise | string> { + const url = registryUrl(packageName); + + try { + const res = await deps.fetchFn(url); + + if (!res.ok) { + return `npm registry returned HTTP ${String(res.status)} for ${packageName}.`; + } + + const json = await res.json(); + + return isRecord(json) ? json : `npm registry returned malformed metadata.`; + } catch (err) { + const message = err instanceof Error ? err.message : "unknown error"; + + return `npm registry request failed for ${packageName}: ${message}`; + } +} + +async function realPackageFetch(url: string): Promise { + return fetch(url, { + headers: { + accept: "application/json", + "user-agent": "tsforge-package-info/1.0 (+keyless)", + }, + }); +} + +const DEFAULT_DEPS: IPackageInfoDeps = { fetchFn: realPackageFetch }; + +export async function doPackageInfo( + args: Record, + ctx: IToolContext, + deps: IPackageInfoDeps = DEFAULT_DEPS +): Promise { + const raw = str(args, "package").trim(); + const packageName = packageNameFromSpec(raw); + + if (packageName === null) { + return reject( + ctx, + "package_info", + "package_info: `package` must be one plain npm package name, optionally @versioned." + ); + } + + ctx.report({ + kind: "tool", + task: ctx.task, + message: `↳ package_info ${packageName}`, + }); + + const manifest = await fetchManifest(packageName, deps); + + if (typeof manifest === "string") { + return `package_info: ${manifest}`; + } + + return truncate( + formatPackageInfo(manifest, versionFromSpec(raw)), + maxChars(args) + ); +} + +async function readIfExists(path: string): Promise { + const file = Bun.file(path); + + if (!(await file.exists())) { + return null; + } + + return file.text(); +} + +/** Locate an installed package's directory, walking up parent `node_modules` + * so hoisted monorepo deps resolve too. Uses the standard node algorithm (walk + * ancestors) rather than `require.resolve(pkg + "/package.json")`, which throws + * for packages whose `exports` map blocks the package.json subpath. Falls back + * to the local `node_modules` path when nothing is found (→ no local docs). */ +async function resolvePackageRoot( + cwd: string, + packageName: string +): Promise { + const segments = packageName.split("/"); + let dir = resolve(cwd); + + for (;;) { + const candidate = join(dir, "node_modules", ...segments); + + if (await Bun.file(join(candidate, "package.json")).exists()) { + return candidate; + } + + const parent = dirname(dir); + + if (parent === dir) { + return join(cwd, "node_modules", ...segments); + } + + dir = parent; + } +} + +/** Resolve `candidate` against `root`, returning the absolute path only if it + * stays inside `root`. The `types` field comes from an installed package's own + * package.json — a hostile dep could set it to `../../../etc/passwd` to have an + * arbitrary file read and disclosed to the agent — so clamp it to the package. */ +function resolveWithin(root: string, candidate: string): string | null { + const target = resolve(root, candidate); + const rel = relative(root, target); + + if (rel.length === 0 || (!rel.startsWith("..") && !isAbsolute(rel))) { + return target; + } + + return null; +} + +async function localDocs( + cwd: string, + packageName: string +): Promise { + const root = await resolvePackageRoot(cwd, packageName); + const pkgText = await readIfExists(join(root, "package.json")); + const parts: string[] = []; + + if (pkgText !== null) { + parts.push( + `# ${packageName} package.json`, + "```json", + pkgText.trim(), + "```" + ); + } + + for (const readme of ["README.md", "readme.md", "README", "README.txt"]) { + const content = await readIfExists(join(root, readme)); + + if (content !== null && content.trim().length > 0) { + parts.push(`# ${packageName} ${readme}`, content.trim()); + break; + } + } + + if (pkgText !== null) { + try { + const parsed: unknown = JSON.parse(pkgText); + + if (isRecord(parsed) && typeof parsed.types === "string") { + const typesPath = resolveWithin(root, parsed.types); + const types = typesPath === null ? null : await readIfExists(typesPath); + + if (types !== null && types.trim().length > 0) { + parts.push( + `# ${packageName} ${parsed.types}`, + "```ts", + types.trim(), + "```" + ); + } + } + } catch { + // Package docs are best-effort; malformed local package.json is ignored. + } + } + + if (parts.length === 0) { + return null; + } + + return { source: "local", content: parts.join("\n\n") }; +} + +function docsSource( + args: Record +): "auto" | "local" | "registry" | null { + const source = args.source; + + if (source === undefined || source === null || source === "") { + return "auto"; + } + + if (source === "auto" || source === "local" || source === "registry") { + return source; + } + + return null; +} + +function registryDocs( + manifest: Record, + packageName: string, + version: string | null +): IPackageDocHit { + const selected = selectedVersion(manifest, version); + const details = versionRecord(manifest, selected); + const homepage = firstNonEmpty( + details === null ? "" : stringProp(details, "homepage"), + stringProp(manifest, "homepage") + ); + const repo = firstNonEmpty( + details === null ? "" : repositoryUrl(details.repository), + repositoryUrl(manifest.repository) + ); + const readme = stringProp(manifest, "readme"); + const lines = [ + `# ${packageName} docs`, + `source: npm registry (${registryRoot()})`, + `version: ${selected.length > 0 ? selected : "(unknown)"}`, + homepage.length > 0 ? `homepage: ${homepage}` : "", + repo.length > 0 ? `repository: ${repo}` : "", + readme.length > 0 ? readme : "(registry metadata did not include a README)", + ].filter((line) => line.length > 0); + + return { source: "registry", content: lines.join("\n\n") }; +} + +export async function doPackageDocs( + args: Record, + ctx: IToolContext, + deps: IPackageInfoDeps = DEFAULT_DEPS +): Promise { + const raw = str(args, "package").trim(); + const packageName = packageNameFromSpec(raw); + + if (packageName === null) { + return reject( + ctx, + "package_docs", + "package_docs: `package` must be one plain npm package name, optionally @versioned." + ); + } + + const source = docsSource(args); + + if (source === null) { + return reject( + ctx, + "package_docs", + "package_docs: `source` must be `auto`, `local`, or `registry`." + ); + } + + ctx.report({ + kind: "tool", + task: ctx.task, + message: `↳ package_docs ${packageName}`, + }); + + if (source === "auto" || source === "local") { + const local = await localDocs(ctx.cwd, packageName); + + if (local !== null) { + return truncate( + `package_docs: source=${local.source}\n\n${local.content}`, + maxChars(args) + ); + } + + if (source === "local") { + return `package_docs: no local docs found for ${packageName} under node_modules.`; + } + } + + const manifest = await fetchManifest(packageName, deps); + + if (typeof manifest === "string") { + return `package_docs: ${manifest}`; + } + + const hit = registryDocs(manifest, packageName, versionFromSpec(raw)); + + return truncate( + `package_docs: source=${hit.source}\n\n${hit.content}`, + maxChars(args) + ); +} diff --git a/packages/core/src/loop/tools/web-browse.ts b/packages/core/src/loop/tools/web-browse.ts new file mode 100644 index 0000000..e0e2357 --- /dev/null +++ b/packages/core/src/loop/tools/web-browse.ts @@ -0,0 +1,210 @@ +import type { chromium as Chromium } from "playwright"; +import { reject, str, type IToolContext } from "./tool-context"; +import { validateFetchUrl } from "./web-fetch"; + +const DEFAULT_MAX_CHARS = 10_000; +const MAX_ALLOWED_CHARS = 40_000; +const MAX_LINKS = 12; + +export interface IBrowseLink { + text: string; + href: string; +} + +export interface IBrowsePage { + goto( + url: string, + options: { waitUntil: "domcontentloaded"; timeout: number } + ): Promise; + title(): Promise; + url(): string; + waitForTimeout(ms: number): Promise; + evaluate(fn: () => T): Promise; + close(): Promise; + route( + pattern: string, + handler: (route: IBrowseRoute) => Promise | void + ): Promise; +} + +export interface IBrowseRoute { + request(): { url(): string }; + continue(): Promise; + abort(): Promise; +} + +export interface IBrowseBrowser { + newPage(): Promise; + close(): Promise; +} + +export interface IWebBrowseDeps { + launchBrowser: () => Promise; +} + +async function loadBrowser(): Promise { + let chromium: typeof Chromium; + + try { + chromium = (await import("playwright")).chromium; + } catch { + return null; + } + + return chromium.launch({ args: ["--no-sandbox"] }); +} + +function maxChars(args: Record): number { + const value = args.maxChars; + + if (typeof value === "number" && Number.isFinite(value) && value > 0) { + return Math.min(Math.floor(value), MAX_ALLOWED_CHARS); + } + + return DEFAULT_MAX_CHARS; +} + +function waitMs(args: Record): number { + const value = args.waitMs; + + if (typeof value === "number" && Number.isFinite(value) && value >= 0) { + return Math.min(Math.floor(value), 10_000); + } + + return 750; +} + +function truncate(content: string, max: number): string { + const trimmed = content.trim(); + + if (trimmed.length <= max) { + return trimmed; + } + + return `${trimmed.slice(0, max)}\n\n...[truncated ${String(trimmed.length - max)} chars - raise maxChars to read more]`; +} + +async function installRequestGuard(page: IBrowsePage): Promise { + await page.route("**/*", async (route) => { + if (validateFetchUrl(route.request().url()) === null) { + await route.abort(); + + return; + } + + await route.continue(); + }); +} + +function renderLinks(links: readonly IBrowseLink[]): string { + if (links.length === 0) { + return "(none)"; + } + + return links + .slice(0, MAX_LINKS) + .map((link, index) => { + const label = link.text.length > 0 ? link.text : link.href; + + return `${String(index + 1)}. ${label}\n ${link.href}`; + }) + .join("\n"); +} + +async function visibleText(page: IBrowsePage): Promise { + // document.body can be absent on a blank/failed load; the index lookup is + // genuinely nullable, so evaluate returns "" instead of throwing a TypeError. + return page.evaluate( + () => document.getElementsByTagName("body")[0]?.innerText ?? "" + ); +} + +async function visibleLinks(page: IBrowsePage): Promise { + return page.evaluate(() => { + const out: IBrowseLink[] = []; + + for (const anchor of document.querySelectorAll("a")) { + const href = anchor.href; + + if (href.length === 0) { + continue; + } + + out.push({ + text: anchor.textContent.replace(/\s+/gu, " ").trim(), + href, + }); + } + + return out; + }); +} + +export async function doWebBrowse( + args: Record, + ctx: IToolContext, + deps: IWebBrowseDeps = DEFAULT_DEPS +): Promise { + const url = validateFetchUrl(str(args, "url")); + + if (url === null) { + return reject( + ctx, + "web_browse", + "web_browse: `url` must be an absolute http(s) URL to a public host." + ); + } + + ctx.report({ + kind: "tool", + task: ctx.task, + message: `↳ web_browse ${url.href}`, + }); + + const browser = await deps.launchBrowser(); + + if (browser === null) { + return ( + "web_browse: Playwright is not installed or Chromium is unavailable. " + + "Use web_fetch for static pages, or install Playwright locally to enable browser browsing." + ); + } + + try { + const page = await browser.newPage(); + + try { + await installRequestGuard(page); + await page.goto(url.href, { + waitUntil: "domcontentloaded", + timeout: 20_000, + }); + await page.waitForTimeout(waitMs(args)); + + const title = await page.title(); + const text = await visibleText(page); + const links = await visibleLinks(page); + const body = [ + `# ${title.length > 0 ? title : page.url()}`, + `url: ${page.url()}`, + "", + truncate(text, maxChars(args)), + "", + "## Links", + renderLinks(links), + ].join("\n"); + + return body.trim(); + } finally { + await page.close(); + } + } catch (err) { + const message = err instanceof Error ? err.message : "unknown error"; + + return `web_browse: failed to browse ${url.href} - ${message}`; + } finally { + await browser.close(); + } +} + +const DEFAULT_DEPS: IWebBrowseDeps = { launchBrowser: loadBrowser }; diff --git a/packages/core/src/loop/tools/web-search.ts b/packages/core/src/loop/tools/web-search.ts index 600845b..5cc2aa4 100644 --- a/packages/core/src/loop/tools/web-search.ts +++ b/packages/core/src/loop/tools/web-search.ts @@ -1,5 +1,6 @@ import { isRecord, isArray } from "../../lib/guards"; import { reject, str, type IToolContext } from "./tool-context"; +import { validateFetchUrl } from "./web-fetch"; /** DuckDuckGo's no-JS HTML endpoint — free, keyless, and returns plain markup we * can parse. The default backend so web search works out of the box with zero @@ -7,7 +8,12 @@ import { reject, str, type IToolContext } from "./tool-context"; * via TSFORGE_SEARXNG_URL instead. */ const DDG_ENDPOINT = "https://html.duckduckgo.com/html/"; -const MAX_RESULTS = 8; +const DEFAULT_MAX_RESULTS = 8; +const MAX_ALLOWED_RESULTS = 20; +const MAX_DOMAINS = 5; + +export type WebSearchRecency = "day" | "month" | "year"; +type WebSearchBackend = "duckduckgo" | "searxng"; export interface ISearchResult { title: string; @@ -25,6 +31,34 @@ export interface IWebSearchDeps { fetchFn: (url: string) => Promise; } +function parseRecency(value: unknown): WebSearchRecency | null { + if (value === "day" || value === "month" || value === "year") { + return value; + } + + return null; +} + +function recencyArg(args: Record): WebSearchRecency | null { + const value = args.recency; + + if (value === undefined || value === null || value === "") { + return null; + } + + return parseRecency(value); +} + +function maxResults(args: Record): number { + const value = args.maxResults; + + if (typeof value === "number" && Number.isFinite(value) && value > 0) { + return Math.min(Math.floor(value), MAX_ALLOWED_RESULTS); + } + + return DEFAULT_MAX_RESULTS; +} + function decodeEntities(s: string): string { return s .replace(/&/g, "&") @@ -57,6 +91,98 @@ function decodeDdgHref(href: string): string { return href.startsWith("//") ? `https:${href}` : href; } +function normalizeDomain(raw: string): string | null { + const trimmed = raw.trim().replace(/^site:/iu, ""); + + if (trimmed.length === 0) { + return null; + } + + let host: string; + + try { + host = /^[a-z][a-z0-9+.-]*:\/\//iu.test(trimmed) + ? new URL(trimmed).hostname + : (trimmed.split(/[/?#]/u)[0] ?? ""); + } catch { + return null; + } + + const normalized = host.toLowerCase().replace(/\.$/u, ""); + + if ( + normalized.length === 0 || + normalized.includes(":") || + !/^[a-z0-9.-]+$/iu.test(normalized) || + normalized.split(".").some((part) => part.length === 0) + ) { + return null; + } + + return validateFetchUrl(`https://${normalized}/`) === null + ? null + : normalized; +} + +function domainsArg(args: Record): string[] | null { + const value = args.domains; + + if (value === undefined || value === null) { + return []; + } + + const rawDomains: string[] = []; + + if (typeof value === "string") { + rawDomains.push(value); + } else if (isArray(value)) { + for (const item of value) { + if (typeof item !== "string") { + return null; + } + + rawDomains.push(item); + } + } else { + return null; + } + + const domains: string[] = []; + + for (const rawDomain of rawDomains) { + const domain = normalizeDomain(rawDomain); + + if (domain === null) { + return null; + } + + if (!domains.includes(domain)) { + domains.push(domain); + } + } + + return domains.slice(0, MAX_DOMAINS); +} + +function scopedQuery(query: string, domains: readonly string[]): string { + if (domains.length === 0) { + return query; + } + + const sites = domains.map((domain) => `site:${domain}`); + const first = sites[0]; + + if (first === undefined) { + return query; + } + + if (sites.length === 1) { + return `${query} ${first}`; + } + + return `${query} (${sites.join(" OR ")})`; +} + const ANCHOR_RE = /]*class="result__a"[^>]*>[\s\S]*?<\/a>/g; const SNIPPET_RE = /]*class="result__snippet"[^>]*>([\s\S]*?)<\/a>/g; const HREF_RE = /href="([^"]*)"/; @@ -116,13 +242,36 @@ export function parseSearxngResults(json: unknown): ISearchResult[] { return out; } -export function formatResults(results: readonly ISearchResult[]): string { +export function filterPublicResults( + results: readonly ISearchResult[] +): ISearchResult[] { + const out: ISearchResult[] = []; + const seen = new Set(); + + for (const result of results) { + const url = validateFetchUrl(result.url); + + if (url === null || seen.has(url.href)) { + continue; + } + + seen.add(url.href); + out.push({ ...result, url: url.href }); + } + + return out; +} + +export function formatResults( + results: readonly ISearchResult[], + limit: number = DEFAULT_MAX_RESULTS +): string { if (results.length === 0) { return "no results found."; } return results - .slice(0, MAX_RESULTS) + .slice(0, limit) .map((r, i) => { const head = r.title.length > 0 ? r.title : r.url; const snip = r.snippet.length > 0 ? `\n ${r.snippet}` : ""; @@ -132,20 +281,93 @@ export function formatResults(results: readonly ISearchResult[]): string { .join("\n\n"); } -function buildSearchUrl(query: string): { url: string; searxng: boolean } { - const base = process.env.TSFORGE_SEARXNG_URL; +function ddgRecencyParam(recency: WebSearchRecency | null): string | null { + if (recency === "day") { + return "d"; + } + + if (recency === "month") { + return "m"; + } + + return recency === "year" ? "y" : null; +} + +function configuredSearxngUrl(): string { + return process.env.TSFORGE_SEARXNG_URL?.trim() ?? ""; +} + +function searchBackend(): WebSearchBackend | null { + const configured = process.env.TSFORGE_WEB_SEARCH_BACKEND?.trim(); + + if (configured === "duckduckgo" || configured === "searxng") { + return configured; + } + + if (configured !== undefined && configured.length > 0) { + return null; + } + + return configuredSearxngUrl().length > 0 ? "searxng" : "duckduckgo"; +} + +interface ISearchUrl { + url: string; + searxng: boolean; +} + +interface ISearchUrlError { + error: string; +} + +function buildSearchUrl( + query: string, + recency: WebSearchRecency | null +): ISearchUrl | ISearchUrlError { + const backend = searchBackend(); + const base = configuredSearxngUrl(); + const params = new URLSearchParams({ q: query }); + + if (backend === null) { + return { + error: + "web_search: TSFORGE_WEB_SEARCH_BACKEND must be `duckduckgo` or `searxng`.", + }; + } + + if (backend === "searxng") { + if (base.length === 0) { + return { + error: + "web_search: TSFORGE_WEB_SEARCH_BACKEND=searxng requires TSFORGE_SEARXNG_URL.", + }; + } - if (base !== undefined && base.length > 0) { const root = base.replace(/\/+$/, ""); + params.set("format", "json"); + + if (recency !== null) { + params.set("time_range", recency); + } + return { - url: `${root}/search?q=${encodeURIComponent(query)}&format=json`, + url: `${root}/search?${params.toString()}`, searxng: true, }; } + const url = new URL(DDG_ENDPOINT); + const ddgRecency = ddgRecencyParam(recency); + + url.searchParams.set("q", query); + + if (ddgRecency !== null) { + url.searchParams.set("df", ddgRecency); + } + return { - url: `${DDG_ENDPOINT}?q=${encodeURIComponent(query)}`, + url: url.href, searxng: false, }; } @@ -179,7 +401,36 @@ export async function doWebSearch( ); } - const { url, searxng } = buildSearchUrl(query); + if ( + args.recency !== undefined && + args.recency !== null && + recencyArg(args) === null + ) { + return reject( + ctx, + "web_search", + "web_search: `recency` must be one of `day`, `month`, or `year`." + ); + } + + const domains = domainsArg(args); + + if (domains === null) { + return reject( + ctx, + "web_search", + "web_search: `domains` must be a string or string[] of public hostnames." + ); + } + + const recency = recencyArg(args); + const searchUrl = buildSearchUrl(scopedQuery(query, domains), recency); + + if ("error" in searchUrl) { + return reject(ctx, "web_search", searchUrl.error); + } + + const { url, searxng } = searchUrl; ctx.report({ kind: "tool", @@ -207,7 +458,7 @@ export async function doWebSearch( ? parseSearxngResults(safeJson(text)) : parseDuckDuckGoResults(text); - return formatResults(results); + return formatResults(filterPublicResults(results), maxResults(args)); } async function realSearchFetch(url: string): Promise { diff --git a/packages/core/src/loop/turn.ts b/packages/core/src/loop/turn.ts index 517e913..700ac3d 100644 --- a/packages/core/src/loop/turn.ts +++ b/packages/core/src/loop/turn.ts @@ -32,6 +32,9 @@ import { LSP_TOOLS, WEB_FETCH_TOOL, WEB_SEARCH_TOOL, + WEB_BROWSE_TOOL, + PACKAGE_INFO_TOOL, + PACKAGE_DOCS_TOOL, GIT_CONTEXT_TOOL, } from "../agent"; import { TsService } from "../lsp"; @@ -77,13 +80,24 @@ type AdvertisedTool = | (typeof LSP_TOOLS)[number] | typeof WEB_FETCH_TOOL | typeof WEB_SEARCH_TOOL + | typeof WEB_BROWSE_TOOL + | typeof PACKAGE_INFO_TOOL + | typeof PACKAGE_DOCS_TOOL | typeof GIT_CONTEXT_TOOL; /** Free, local web tools (fetch + search) — advertised only under TSFORGE_WEB so * eval sweeps stay deterministic and offline by default. Available on both * scratch and existing-code runs when enabled (unlike the LSP nav set). */ function webTools(): AdvertisedTool[] { - return flags.webTools() ? [WEB_FETCH_TOOL, WEB_SEARCH_TOOL] : []; + return flags.webTools() + ? [ + WEB_FETCH_TOOL, + WEB_SEARCH_TOOL, + WEB_BROWSE_TOOL, + PACKAGE_INFO_TOOL, + PACKAGE_DOCS_TOOL, + ] + : []; } /** Read-only git introspection — existing-code runs only (greenfield has no diff --git a/packages/core/src/policy/classify.ts b/packages/core/src/policy/classify.ts index accccf8..457362f 100644 --- a/packages/core/src/policy/classify.ts +++ b/packages/core/src/policy/classify.ts @@ -28,8 +28,11 @@ const KIND_BY_TOOL: Readonly> = { [TOOL_NAME.scaffoldWeb]: "write_file", [TOOL_NAME.run]: "shell", [TOOL_NAME.addDependency]: "shell", + [TOOL_NAME.packageInfo]: "network", + [TOOL_NAME.packageDocs]: "network", [TOOL_NAME.webFetch]: "network", [TOOL_NAME.webSearch]: "network", + [TOOL_NAME.webBrowse]: "network", }; /** Extra path-bearing arg keys beyond the file aliases (move's source/target). */ diff --git a/packages/core/tests/execute-tool.test.ts b/packages/core/tests/execute-tool.test.ts index d6f894b..617a1dd 100644 --- a/packages/core/tests/execute-tool.test.ts +++ b/packages/core/tests/execute-tool.test.ts @@ -623,6 +623,33 @@ test("web_search dispatches to its handler (empty query rejected, no network)", expect(r).not.toContain("plan mode"); }); +test("package and browser research tools are permitted in plan mode", async () => { + const ctx = { + cwd: ".", + files: [], + task: "t", + report: () => undefined, + readOnly: true, + }; + const info = await executeTool( + { name: "package_info", arguments: { package: "../x" } }, + ctx + ); + const docs = await executeTool( + { name: "package_docs", arguments: { package: "../x" } }, + ctx + ); + const browse = await executeTool( + { name: "web_browse", arguments: { url: "file:///etc/passwd" } }, + ctx + ); + + expect(info).toContain("package_info"); + expect(docs).toContain("package_docs"); + expect(browse).toContain("web_browse"); + expect(`${info}\n${docs}\n${browse}`).not.toContain("plan mode"); +}); + test("edit not-found on a file the model AUTHORED offers the create-rewrite escape hatch", async () => { // F24: the model painted its own service file into a corner (stale anchors + // too-large edits) and thrashed ~20 turns because the not-found message said diff --git a/packages/core/tests/package-info.test.ts b/packages/core/tests/package-info.test.ts new file mode 100644 index 0000000..ad7aad0 --- /dev/null +++ b/packages/core/tests/package-info.test.ts @@ -0,0 +1,272 @@ +import { test, expect } from "bun:test"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + doPackageDocs, + doPackageInfo, + packageNameFromSpec, + type IPackageInfoDeps, +} from "../src/loop/tools/package-info"; +import type { IToolContext } from "../src/loop/tools/tool-context"; + +const MANIFEST = { + name: "zod", + description: "schema validation", + "dist-tags": { latest: "4.2.0", next: "4.3.0-beta.1" }, + versions: { + "4.1.0": { + version: "4.1.0", + license: "MIT", + }, + "4.2.0": { + version: "4.2.0", + license: "MIT", + peerDependencies: { typescript: ">=5" }, + homepage: "https://zod.dev", + }, + }, + repository: { url: "git+https://github.com/colinhacks/zod.git" }, + readme: "# Zod\n\nCurrent docs.", +}; + +function ctx(cwd = "."): IToolContext { + return { cwd, files: [], task: "t", report: () => undefined }; +} + +function deps(json: unknown, requested: string[] = []): IPackageInfoDeps { + return { + fetchFn: async (url) => { + requested.push(url); + + return { + ok: true, + status: 200, + json: async () => json, + }; + }, + }; +} + +test("packageNameFromSpec accepts scoped/unscoped optional versions", () => { + expect(packageNameFromSpec("zod")).toBe("zod"); + expect(packageNameFromSpec("zod@4")).toBe("zod"); + expect(packageNameFromSpec("@tanstack/react-query")).toBe( + "@tanstack/react-query" + ); + expect(packageNameFromSpec("@tanstack/react-query@5")).toBe( + "@tanstack/react-query" + ); + expect(packageNameFromSpec("--flag")).toBeNull(); +}); + +test("doPackageInfo fetches npm metadata and formats latest package details", async () => { + const requested: string[] = []; + const out = await doPackageInfo( + { package: "zod", maxChars: 5000 }, + ctx(), + deps(MANIFEST, requested) + ); + + expect(requested[0]).toContain("registry.npmjs.org/zod"); + expect(out).toContain("selected: 4.2.0"); + expect(out).toContain("latest: 4.2.0"); + expect(out).toContain("peerDependencies: typescript"); + expect(out).toContain("https://zod.dev"); +}); + +test("doPackageInfo honors an explicit version in the package spec", async () => { + const out = await doPackageInfo( + { package: "zod@4.1.0" }, + ctx(), + deps(MANIFEST) + ); + + expect(out).toContain("selected: 4.1.0"); +}); + +test("doPackageDocs prefers local node_modules docs in auto mode", async () => { + const dir = await mkdtemp(join(tmpdir(), "tsforge-pkgdocs-")); + + try { + const root = join(dir, "node_modules", "zod"); + + await mkdir(root, { recursive: true }); + await writeFile( + join(root, "package.json"), + JSON.stringify({ name: "zod", version: "4.2.0", types: "index.d.ts" }) + ); + await writeFile(join(root, "README.md"), "# Local Zod docs\n"); + await writeFile( + join(root, "index.d.ts"), + "export declare const z: string;\n" + ); + + const out = await doPackageDocs( + { package: "zod" }, + ctx(dir), + deps(MANIFEST) + ); + + expect(out).toContain("source=local"); + expect(out).toContain("Local Zod docs"); + expect(out).toContain("index.d.ts"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("doPackageDocs finds a package hoisted to an ancestor node_modules", async () => { + // Monorepo layout: deps hoist to the repo-root node_modules, but the model + // runs from a nested workspace dir. The walk-up must still find them. + const dir = await mkdtemp(join(tmpdir(), "tsforge-pkgdocs-")); + + try { + const hoisted = join(dir, "node_modules", "zod"); + const workspace = join(dir, "packages", "app"); + + await mkdir(hoisted, { recursive: true }); + await mkdir(workspace, { recursive: true }); + await writeFile( + join(hoisted, "package.json"), + JSON.stringify({ name: "zod", version: "4.2.0" }) + ); + await writeFile(join(hoisted, "README.md"), "# Hoisted Zod docs\n"); + + const out = await doPackageDocs( + { package: "zod" }, + ctx(workspace), + deps(MANIFEST) + ); + + expect(out).toContain("source=local"); + expect(out).toContain("Hoisted Zod docs"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("doPackageDocs refuses a `types` path that escapes the package root", async () => { + // A hostile installed dep could set types to `../../secret` to have an + // arbitrary file read and disclosed to the agent — the read must be clamped. + const dir = await mkdtemp(join(tmpdir(), "tsforge-pkgdocs-")); + + try { + const root = join(dir, "node_modules", "evil"); + + await mkdir(root, { recursive: true }); + await writeFile( + join(root, "package.json"), + JSON.stringify({ name: "evil", version: "1.0.0", types: "../../secret" }) + ); + await writeFile(join(root, "README.md"), "# Evil docs\n"); + await writeFile(join(dir, "secret"), "TOP SECRET CONTENTS\n"); + + const out = await doPackageDocs( + { package: "evil" }, + ctx(dir), + deps(MANIFEST) + ); + + expect(out).toContain("source=local"); + expect(out).toContain("Evil docs"); + // The escaping `types` target is never read, so its contents never leak. + expect(out).not.toContain("TOP SECRET"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("doPackageDocs falls back to registry README when local docs are absent", async () => { + const out = await doPackageDocs( + { package: "zod", source: "registry" }, + ctx(), + deps(MANIFEST) + ); + + expect(out).toContain("source=registry"); + expect(out).toContain("Current docs"); +}); + +test("package tools reject invalid args without touching the network", async () => { + let called = false; + const noNetwork: IPackageInfoDeps = { + fetchFn: async () => { + called = true; + + throw new Error("should not be called"); + }, + }; + + const info = await doPackageInfo({ package: "../x" }, ctx(), noNetwork); + const docs = await doPackageDocs( + { package: "zod", source: "somewhere" }, + ctx(), + noNetwork + ); + + expect(called).toBe(false); + expect(info).toContain("package_info"); + expect(docs).toContain("source"); +}); + +test("a requested major/range resolves to a concrete version with its deps", async () => { + // `react@19` indexes nothing in the manifest (keyed by exact version), so + // without resolution versionRecord misses and dependency lists come back + // empty. The major must resolve to the highest matching concrete version. + const manifest = { + name: "react", + "dist-tags": { latest: "19.1.0" }, + versions: { + "18.3.1": { + version: "18.3.1", + dependencies: { "loose-envify": "^1.1.0" }, + }, + "19.0.0": { version: "19.0.0" }, + "19.1.0": { + version: "19.1.0", + peerDependencies: { "react-dom": "19.1.0" }, + }, + }, + }; + + const major = await doPackageInfo( + { package: "react@19" }, + ctx(), + deps(manifest) + ); + const range = await doPackageInfo( + { package: "react@^19.0.0" }, + ctx(), + deps(manifest) + ); + + expect(major).toContain("selected: 19.1.0"); + expect(major).toContain("peerDependencies: react-dom"); + // A caret range resolves to the highest compatible concrete version too. + expect(range).toContain("selected: 19.1.0"); + // The "1" prefix must NOT match across a major boundary into 18.x. + expect(major).not.toContain("selected: 18"); +}); + +test("recent versions and no-dist-tag latest sort by semver, not lexically", async () => { + // Object key order is unspecified and a lexical sort misorders these + // (1.10.0 < 1.9.0 as strings). With no dist-tags, latest must be the highest + // semver, and "recent" must end on it. + const manifest = { + name: "pkg", + versions: { + "1.9.0": { version: "1.9.0" }, + "1.10.0": { version: "1.10.0" }, + "1.2.0": { version: "1.2.0" }, + "1.10.0-rc.1": { version: "1.10.0-rc.1" }, + }, + }; + + const out = await doPackageInfo({ package: "pkg" }, ctx(), deps(manifest)); + + // No dist-tags → fallback latest is the highest stable semver, not "1.9.0". + expect(out).toContain("selected: 1.10.0"); + // Recent list is semver-ascending and the prerelease sorts below its release. + expect(out).toContain("recent: 1.2.0, 1.9.0, 1.10.0-rc.1, 1.10.0"); +}); diff --git a/packages/core/tests/policy-evaluation.test.ts b/packages/core/tests/policy-evaluation.test.ts index 0097500..f9ae394 100644 --- a/packages/core/tests/policy-evaluation.test.ts +++ b/packages/core/tests/policy-evaluation.test.ts @@ -48,8 +48,11 @@ describe("classifyAction", () => { ["scaffold_ui", "write_file"], ["run", "shell"], ["add_dependency", "shell"], + ["package_info", "network"], + ["package_docs", "network"], ["web_fetch", "network"], ["web_search", "network"], + ["web_browse", "network"], ]; for (const [name, kind] of cases) { diff --git a/packages/core/tests/prompt-conventions.test.ts b/packages/core/tests/prompt-conventions.test.ts index e29da83..3d4a107 100644 --- a/packages/core/tests/prompt-conventions.test.ts +++ b/packages/core/tests/prompt-conventions.test.ts @@ -1,4 +1,4 @@ -import { describe, test, expect } from "bun:test"; +import { describe, test, expect, afterEach } from "bun:test"; import { buildChatSystem, buildSystemPrompt, @@ -10,6 +10,10 @@ const bare = resolveConventions({ interfaces: "bare-pascal-case" }); const iprefix = resolveConventions({ interfaces: "i-prefix" }); const off = resolveConventions({ interfaces: "off" }); +afterEach(() => { + delete process.env.TSFORGE_WEB; +}); + describe("system prompt reflects interface convention", () => { test("i-prefix tells the model to use the I prefix", () => { const p = buildSystemPrompt(false, undefined, iprefix); @@ -49,6 +53,28 @@ describe("chat prompt reflects interface convention", () => { }); }); +describe("web research guidance", () => { + test("web guidance is absent unless TSFORGE_WEB is enabled", () => { + expect(buildSystemPrompt(false, undefined, iprefix)).not.toContain( + "WEB RESEARCH" + ); + expect(buildChatSystem(iprefix)).not.toContain("Web tools are enabled"); + }); + + test("web guidance names the browsing tools and freshness controls", () => { + process.env.TSFORGE_WEB = "1"; + + const system = buildSystemPrompt(false, undefined, iprefix); + const chat = buildChatSystem(iprefix); + + expect(system).toContain("WEB RESEARCH"); + expect(system).toContain("web_search"); + expect(system).toContain("recency"); + expect(chat).toContain("web_fetch"); + expect(chat).toContain("maxResults"); + }); +}); + describe("TDD guidance reflects test layout", () => { test("co-located vs mirrored phrasing", () => { expect( diff --git a/packages/core/tests/tools-gating.test.ts b/packages/core/tests/tools-gating.test.ts index 9aa98c4..5c82c27 100644 --- a/packages/core/tests/tools-gating.test.ts +++ b/packages/core/tests/tools-gating.test.ts @@ -66,14 +66,20 @@ test("web tools are absent unless TSFORGE_WEB=1", () => { expect(n).not.toContain("web_fetch"); expect(n).not.toContain("web_search"); + expect(n).not.toContain("web_browse"); + expect(n).not.toContain("package_info"); + expect(n).not.toContain("package_docs"); }); -test("TSFORGE_WEB=1 exposes both free web tools (no key required)", () => { +test("TSFORGE_WEB=1 exposes keyless web/package research tools", () => { process.env.TSFORGE_WEB = "1"; const n = names(toolsFor(true)); expect(n).toContain("web_fetch"); expect(n).toContain("web_search"); + expect(n).toContain("web_browse"); + expect(n).toContain("package_info"); + expect(n).toContain("package_docs"); }); test("web tools are available on scratch tasks too when enabled", () => { @@ -82,4 +88,7 @@ test("web tools are available on scratch tasks too when enabled", () => { expect(n).toContain("web_fetch"); expect(n).toContain("web_search"); + expect(n).toContain("web_browse"); + expect(n).toContain("package_info"); + expect(n).toContain("package_docs"); }); diff --git a/packages/core/tests/web-browse.test.ts b/packages/core/tests/web-browse.test.ts new file mode 100644 index 0000000..aaf1d2c --- /dev/null +++ b/packages/core/tests/web-browse.test.ts @@ -0,0 +1,31 @@ +import { test, expect } from "bun:test"; +import { doWebBrowse, type IWebBrowseDeps } from "../src/loop/tools/web-browse"; +import type { IToolContext } from "../src/loop/tools/tool-context"; + +function ctx(): IToolContext { + return { cwd: ".", files: [], task: "t", report: () => undefined }; +} + +test("doWebBrowse rejects invalid URLs without launching a browser", async () => { + let launched = false; + const deps: IWebBrowseDeps = { + launchBrowser: async () => { + launched = true; + + return null; + }, + }; + const out = await doWebBrowse({ url: "file:///etc/passwd" }, ctx(), deps); + + expect(launched).toBe(false); + expect(out).toContain("web_browse"); +}); + +test("doWebBrowse explains when local Playwright is unavailable", async () => { + const out = await doWebBrowse({ url: "https://example.com" }, ctx(), { + launchBrowser: async () => null, + }); + + expect(out).toContain("Playwright"); + expect(out).toContain("web_fetch"); +}); diff --git a/packages/core/tests/web-search.test.ts b/packages/core/tests/web-search.test.ts index 69b8baa..bb417f6 100644 --- a/packages/core/tests/web-search.test.ts +++ b/packages/core/tests/web-search.test.ts @@ -2,6 +2,7 @@ import { test, expect, afterEach } from "bun:test"; import { parseDuckDuckGoResults, formatResults, + filterPublicResults, doWebSearch, type IWebSearchDeps, } from "../src/loop/tools/web-search"; @@ -34,6 +35,7 @@ const deps = (over: Partial): IWebSearchDeps => ({ afterEach(() => { delete process.env.TSFORGE_SEARXNG_URL; + delete process.env.TSFORGE_WEB_SEARCH_BACKEND; }); test("parseDuckDuckGoResults extracts title, decoded url, and snippet", () => { @@ -56,10 +58,33 @@ test("formatResults lists each result with title, url, and snippet", () => { expect(out).toContain("about foo"); }); +test("formatResults honors a caller-supplied result limit", () => { + const out = formatResults( + [ + { title: "One", url: "https://one.com", snippet: "" }, + { title: "Two", url: "https://two.com", snippet: "" }, + ], + 1 + ); + + expect(out).toContain("One"); + expect(out).not.toContain("Two"); +}); + test("formatResults reports when there are no results", () => { expect(formatResults([]).toLowerCase()).toContain("no results"); }); +test("filterPublicResults drops unsafe and duplicate URLs", () => { + const out = filterPublicResults([ + { title: "public", url: "https://example.com/a", snippet: "" }, + { title: "private", url: "http://localhost:3000", snippet: "" }, + { title: "dupe", url: "https://example.com/a", snippet: "" }, + ]); + + expect(out.map((r) => r.title)).toEqual(["public"]); +}); + test("doWebSearch rejects an empty query without touching the network", async () => { let called = false; const r = await doWebSearch( @@ -85,6 +110,64 @@ test("doWebSearch returns formatted results from the free DuckDuckGo backend (no expect(r).toContain("https://other.com"); }); +test("doWebSearch passes recency and domain scope to DuckDuckGo", async () => { + let requested = ""; + + await doWebSearch( + { + query: "typescript decorators", + recency: "month", + domains: ["typescriptlang.org", "devblogs.microsoft.com/typescript"], + }, + ctx(), + deps({ + fetchFn: async (url) => { + requested = url; + + return { ok: true, status: 200, text: async () => DDG_HTML }; + }, + }) + ); + + const url = new URL(requested); + const query = url.searchParams.get("q") ?? ""; + + expect(url.searchParams.get("df")).toBe("m"); + expect(query).toContain("typescript decorators"); + expect(query).toContain("site:typescriptlang.org"); + expect(query).toContain("site:devblogs.microsoft.com"); +}); + +test("doWebSearch rejects invalid recency and domains without touching the network", async () => { + let called = false; + const badRecency = await doWebSearch( + { query: "typescript", recency: "week" }, + ctx(), + deps({ + fetchFn: async () => { + called = true; + + throw new Error("should not be called"); + }, + }) + ); + const badDomain = await doWebSearch( + { query: "typescript", domains: "localhost" }, + ctx(), + deps({ + fetchFn: async () => { + called = true; + + throw new Error("should not be called"); + }, + }) + ); + + expect(called).toBe(false); + expect(badRecency).toContain("recency"); + expect(badDomain).toContain("domains"); +}); + test("doWebSearch reports an HTTP error status gracefully", async () => { const r = await doWebSearch( { query: "x" }, @@ -120,3 +203,90 @@ test("doWebSearch routes to a self-hosted SearXNG instance when TSFORGE_SEARXNG_ expect(r).toContain("https://s.com"); expect(r).toContain("searx snippet"); }); + +test("doWebSearch can require SearXNG and fail closed instead of falling back to DuckDuckGo", async () => { + process.env.TSFORGE_WEB_SEARCH_BACKEND = "searxng"; + let called = false; + const r = await doWebSearch( + { query: "typescript" }, + ctx(), + deps({ + fetchFn: async () => { + called = true; + + throw new Error("should not be called"); + }, + }) + ); + + expect(called).toBe(false); + expect(r).toContain("TSFORGE_SEARXNG_URL"); +}); + +test("doWebSearch rejects an invalid search backend before network", async () => { + process.env.TSFORGE_WEB_SEARCH_BACKEND = "local"; + let called = false; + const r = await doWebSearch( + { query: "typescript" }, + ctx(), + deps({ + fetchFn: async () => { + called = true; + + throw new Error("should not be called"); + }, + }) + ); + + expect(called).toBe(false); + expect(r).toContain("TSFORGE_WEB_SEARCH_BACKEND"); +}); + +test("doWebSearch can force DuckDuckGo even when a SearXNG URL is present", async () => { + process.env.TSFORGE_SEARXNG_URL = "http://localhost:8888"; + process.env.TSFORGE_WEB_SEARCH_BACKEND = "duckduckgo"; + let requested = ""; + + await doWebSearch( + { query: "typescript" }, + ctx(), + deps({ + fetchFn: async (url) => { + requested = url; + + return { ok: true, status: 200, text: async () => DDG_HTML }; + }, + }) + ); + + expect(requested).toContain("duckduckgo.com"); + expect(requested).not.toContain("localhost:8888"); +}); + +test("doWebSearch passes recency to SearXNG and applies maxResults", async () => { + process.env.TSFORGE_SEARXNG_URL = "http://localhost:8888"; + let requested = ""; + const json = JSON.stringify({ + results: [ + { title: "Fresh", url: "https://fresh.com", content: "new" }, + { title: "Old", url: "https://old.com", content: "old" }, + ], + }); + const r = await doWebSearch( + { query: "typescript 6", recency: "day", maxResults: 1 }, + ctx(), + deps({ + fetchFn: async (url) => { + requested = url; + + return { ok: true, status: 200, text: async () => json }; + }, + }) + ); + + const url = new URL(requested); + + expect(url.searchParams.get("time_range")).toBe("day"); + expect(r).toContain("Fresh"); + expect(r).not.toContain("Old"); +});