Custom proxy config for crawl4ai + Agent instructions

CLAUDE.md

@@ -29,8 +29,10 @@ Follow these rules when editing code in this project.
 
 `sibyl` is a CLI web search/crawl tool for AI Agents (`bin: sibyl` → `dist/cli.js`) with a filesystem-based plugin system. Key modules:
 
-- `src/cli.ts` — entry point. Ensures dirs + config exist, loads plugins, builds a `PluginContext` (`buildPluginContext`), and dispatches commands (`search`, `fetch`, `ask`, `help`/`--help`/`-h`, `version`/`--version`). `search`, `fetch`, and `ask` are wired up via the async `handleSearch`/`handleFetch`/`handleAsk` helpers (awaited by `main`), each passing the context as the last arg to the selected plugin's `fn`. The `fetch` command prints the fetch plugin's output directly — the CLI doesn't dispatch a separate `parse` step, but a fetch plugin may itself run the configured parse plugin via `context.configuredPlugins.parse` (`builtin-brightdata-fetch`, `builtin-crawl4ai-fetch`, and `builtin-alterlab-fetch` do; `builtin-firecrawl-fetch` does only in its raw-HTML mode; `builtin-exa-fetch` returns content as-is). The `ask` command (`sibyl ask <url> <question>`) passes the URL as the ask plugin's first arg; analogously, an ask plugin may itself fetch that URL via `context.configuredPlugins.fetch` before answering (`builtin-ai-ask` does). `main` is exported and only auto-runs when the file is the actual CLI entry (`import.meta.url` vs `process.argv[1]` guard), so tests can import it without side effects.
+- `src/cli.ts` — entry point. Ensures dirs + config exist, loads plugins, builds a `PluginContext` (`buildPluginContext`), and dispatches commands (`search`, `fetch`, `ask`, `setup`, `help`/`--help`/`-h`, `version`/`--version`). `search`, `fetch`, and `ask` are wired up via the async `handleSearch`/`handleFetch`/`handleAsk` helpers (awaited by `main`), each passing the context as the last arg to the selected plugin's `fn`. The `fetch` command prints the fetch plugin's output directly — the CLI doesn't dispatch a separate `parse` step, but a fetch plugin may itself run the configured parse plugin via `context.configuredPlugins.parse` (`builtin-brightdata-fetch`, `builtin-crawl4ai-fetch`, and `builtin-alterlab-fetch` do; `builtin-firecrawl-fetch` does only in its raw-HTML mode; `builtin-exa-fetch` returns content as-is). The `ask` command (`sibyl ask <url> <question>`) passes the URL as the ask plugin's first arg; analogously, an ask plugin may itself fetch that URL via `context.configuredPlugins.fetch` before answering (`builtin-ai-ask` does). The `setup` command is handled synchronously by `runSetup(rest)` (`src/setup-command.ts`) and uses neither plugins nor the `PluginContext` (see The `setup` command). `main` is exported and only auto-runs when the file is the actual CLI entry (`import.meta.url` vs `process.argv[1]` guard), so tests can import it without side effects.
 - `src/setup.ts` — ensures `~/.sibyl` and `~/.sibyl/plugins` exist, and loads/creates/validates `~/.sibyl/config.json` (all on every invocation).
+- `src/setup-command.ts` — implements the `setup` command (`runSetup`): installs the instructions doc into agent tools' global instruction files (see The `setup` command). Distinct from `setup.ts`, which is config bootstrap.
+- `src/instructions.ts` — the bundled `SIBYL.md` content (`SIBYL_INSTRUCTIONS`) plus the import line, marker, and notice constants used by `setup-command.ts`. Inlined as a string constant (not read from disk) so it ships in `dist`.
 - `src/plugin-loader.ts` — assembles the active plugin set: builtin plugins + external (on-disk) plugins; validates the external ones.
 - `src/plugins/config.ts` — `getBuiltinPlugins()`, the in-repo builtin plugin registry.
 - `src/utils.ts` — pure helpers: `isValidHttpUrl`, `stripSearchResultDatePrefix` (strips localized SERP date prefixes), `collapseBlankLines`, and the search-setting readers `getSearchResultsLimit` / `shouldShowSearchDescription` (see Conventions).
@@ -79,6 +81,18 @@ Shape: `SibylConfig` (`src/@types/sibyl-config.ts`) — `{ plugins: Partial<Reco
 - `validateConfig()` checks each entry's name is a non-empty string; on failure it `console.error`s and `process.exit(1)` (hard exit, not a skip-with-warning like plugin loading).
 - Plugin selection: `loadPlugins()` loads _all_ available plugins (builtins + disk), then `cli.ts` picks the one to run **by name from config** — e.g. the `search` command looks up `config.plugins.search` and finds the loaded plugin whose `type === "search"` and `name` matches. Missing config entry or no matching loaded plugin → `console.error` + non-zero exit.
 
+### The `setup` command (`sibyl setup <targets>`)
+
+`runSetup(args)` (`src/setup-command.ts`) installs the bundled instructions doc into an agent tool's **global** (user-level home-dir) instruction files. It parses per-target flags — at least one is required, else it prints usage and `exit(1)` (also on an unknown flag or an `--other` with no path):
+
+- `--claude` → writes `~/.claude/SIBYL.md` and adds a `@SIBYL.md` import line to `~/.claude/CLAUDE.md`.
+- `--opencode` → writes `~/.config/opencode/SIBYL.md` and adds `~/.config/opencode/SIBYL.md` to the `instructions[]` array in `~/.config/opencode/opencode.json`.
+- `--codex` → embeds the content into `~/.codex/AGENTS.md`.
+- `--antigravity` → embeds the content into `~/.gemini/GEMINI.md`.
+- `--other <file>` (repeatable; `--other=<file>` also accepted) → embeds the content into an arbitrary file.
+
+Two mechanisms: Claude and opencode **reference** a `SIBYL.md` file each target owns; Codex, Antigravity, and `--other` **embed** the content between `<!-- SIBYL:START -->` / `<!-- SIBYL:END -->` markers, followed by a do-not-edit notice line. Everything is idempotent and refreshes in place on re-run — doc files are overwritten, the marker block (and its trailing notice) is regex-replaced, and the `@SIBYL.md` line / opencode entry are added at most once. The content is inlined in `src/instructions.ts` (a bundled string constant) because the `tsc`-only build has no asset-copy step, so a raw `.md` would never reach `dist`.
+
 ## Conventions
 
 - ESM only (`"type": "module"`), Node `>=22`.

README.md

@@ -26,8 +26,7 @@ Get a working setup in a few steps:
 1. Install **Sibyl** globally via NPM:
 
    ```bash
-   # ⚠️ Not yet available on npm
-   npm i -g sibyl
+   npm i -g @postapsis/sibyl
    ```
 
 2. Run a local [SearXNG](https://github.com/searxng/searxng) instance for searching the web:
@@ -65,24 +64,35 @@ Get a working setup in a few steps:
      unclecode/crawl4ai:latest
    ```
 
-4. Run your first search:
+4. Set up your agent to use Sibyl:
+
+   ```bash
+   sibyl setup --claude
+   sibyl setup --codex
+   sibyl setup --opencode
+   sibyl setup --antigravity
+   sibyl setup --other ~/path/to/AGENTS.md
+   ```
+
+5. Run your first search:
 
    ```bash
    sibyl search "how to use react with vite"
    ```
 
-5. Configure your settings!\
+6. Configure your settings!\
    Check the [Configuration](#configuration) section for more details.
 
 ## Commands
 
-| Command              | Description                                                                                                                   |
-| -------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
-| search <query>       | Searches the web <br/>`sibyl search "react vite"`                                                                             |
-| fetch <url>          | Prints the content of a site in token-efficient markdown <br/>`sibyl fetch https://vite.dev/guide`                            |
-| ask <url> <question> | Asks a query using LLM from a site's content <br/>`sibyl ask https://vite.dev/guide "how to start a react project with vite"` |
-| `--help`, `-h`       | Shows help.                                                                                                                   |
-| `--version`          | Shows version.                                                                                                                |
+| Command              | Description                                                                                                                                                                                                                                                                 |
+| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| search <query>       | Searches the web <br/>`sibyl search "react vite"`                                                                                                                                                                                                                           |
+| fetch <url>          | Prints the content of a site in token-efficient markdown <br/>`sibyl fetch https://vite.dev/guide`                                                                                                                                                                          |
+| ask <url> <question> | Asks a query using LLM from a site's content <br/>`sibyl ask https://vite.dev/guide "how to start a react project with vite"`                                                                                                                                               |
+| setup <targets>      | Installs the `SIBYL.md` instructions doc into an agent's global instruction files <br/><br/>`sibyl setup --claude --opencode --codex --antigravity` <br/>Targets: `--claude`, `--opencode`, `--codex`, `--antigravity`, `--other ~/path/to/AGENTS.md` (embed into any file) |
+| `--help`, `-h`       | Shows help.                                                                                                                                                                                                                                                                 |
+| `--version`          | Shows version.                                                                                                                                                                                                                                                              |
 
 ## Configuration
 

docs/CONFIGURATION.md

@@ -78,9 +78,12 @@ Requires a SearXNG instance with the **JSON output format enabled**. See more at
 
 ### Crawl4AI Fetch
 
-| Variable             | Required | Default                  | Description                                                                               |
-| -------------------- | -------- | ------------------------ | ----------------------------------------------------------------------------------------- |
-| `SIBYL_CRAWL4AI_URL` | No       | `http://localhost:11235` | Base URL of a running Crawl4AI server. Sibyl uses the `/crawl` endpoint to fetch the data |
+| Variable                        | Required | Default                  | Description                                                                               |
+| ------------------------------- | -------- | ------------------------ | ----------------------------------------------------------------------------------------- |
+| `SIBYL_CRAWL4AI_URL`            | No       | `http://localhost:11235` | Base URL of a running Crawl4AI server. Sibyl uses the `/crawl` endpoint to fetch the data |
+| `SIBYL_CRAWL4AI_PROXY_SERVER`   | No       | _(none)_                 | Proxy server URL for the crawler. When unset, no proxy is used.                           |
+| `SIBYL_CRAWL4AI_PROXY_USERNAME` | No       | _(none)_                 | Proxy auth username. Sent only when set and a proxy server is configured.                 |
+| `SIBYL_CRAWL4AI_PROXY_PASSWORD` | No       | _(none)_                 | Proxy auth password. Sent only when set and a proxy server is configured.                 |
 
 Requires a Crawl4AI server, e.g., via Docker. See more at [https://hub.docker.com/r/unclecode/crawl4ai](https://hub.docker.com/r/unclecode/crawl4ai)
 

package.json

@@ -1,9 +1,12 @@
 {
-  "name": "sibyl",
+  "name": "@postapsis/sibyl",
   "version": "0.1.0",
   "description": "Give your AI Agent the web, without the bloat — extensible and lightweight by design 🕷️",
   "type": "module",
   "packageManager": "pnpm@11.5.1",
+  "publishConfig": {
+    "access": "public"
+  },
   "main": "dist/cli.js",
   "bin": {
     "sibyl": "dist/cli.js"

src/cli.test.ts

@@ -12,6 +12,10 @@ vi.mock("./plugin-loader.ts", () => ({
   loadPlugins: vi.fn(),
 }));
 
+vi.mock("./setup-command.ts", () => ({
+  runSetup: vi.fn(),
+}));
+
 vi.mock("./exit.ts", () => ({
   exit: vi.fn(() => {
     throw new Error("process.exit");
@@ -22,6 +26,7 @@ import { afterEach, beforeEach, describe, expect, it, vi, type Mock } from "vite
 import { main } from "./cli.ts";
 import { loadOrCreateConfigFile } from "./setup.ts";
 import { loadPlugins } from "./plugin-loader.ts";
+import { runSetup } from "./setup-command.ts";
 import { exit } from "./exit.ts";
 import type {
   AskPlugin,
@@ -160,6 +165,13 @@ describe("dispatch & argument validation", () => {
     expect(console.log).toHaveBeenCalledWith(expect.stringContaining("sibyl - CLI tool"));
     expect(exit).toHaveBeenCalledWith(1);
   });
+
+  it("passes setup flags through to runSetup", async () => {
+    await main(["setup", "--claude", "--codex"]);
+
+    expect(runSetup).toHaveBeenCalledWith(["--claude", "--codex"]);
+    expect(exit).not.toHaveBeenCalled();
+  });
 });
 
 describe("handleSearch", () => {

src/cli.ts

@@ -16,6 +16,7 @@ import type {
 import type { SibylConfig } from "./@types/sibyl-config.ts";
 import { loadPlugins } from "./plugin-loader.ts";
 import { isValidHttpUrl } from "./utils.ts";
+import { runSetup } from "./setup-command.ts";
 import { exit } from "./exit.ts";
 import { pathToFileURL } from "node:url";
 import { realpathSync } from "node:fs";
@@ -82,6 +83,9 @@ export async function main(argv: string[]): Promise<void> {
       await handleAsk(plugins, config, url, query, context);
       break;
     }
+    case "setup":
+      runSetup(rest);
+      break;
     case "--version":
     case "version":
       console.log("sibyl 0.1.0");
@@ -212,13 +216,19 @@ Commands:
   search <query>        Search the web
   fetch <url>           Fetch the content of a URL
   ask <url> <question>  Answer a question using a URL's content via an LLM
+  setup <targets>       Install SIBYL.md into an agent's instruction files
   help                  Show this help
   version               Show version
 
+Setup targets (at least one required):
+  --claude --opencode --codex --antigravity   Install for that agent (global)
+  --other <file>                              Embed instructions into any .MD agent instructions file
+
 Examples:
   sibyl search "react vite"
   sibyl fetch https://vite.dev/guide
   sibyl ask https://vite.dev/guide "how do I start a react project with vite?"
+  sibyl setup --claude
 `);
 }
 

src/instructions.ts

@@ -0,0 +1,45 @@
+/*
+ * Author: Jamius Siam
+ * Since: 14/07/2026
+ */
+
+// The bundled `SIBYL.md` instructions doc installed by `sibyl setup`. Inlined here (rather
+// than read from disk) because the build is `tsc`-only with no asset-copy step, so a raw
+// `.md` file would never reach `dist`. Assembled as a line array joined with newlines so the
+// fenced code block below doesn't need its backticks escaped inside a template literal.
+export const SIBYL_INSTRUCTIONS = [
+  "# Search the web using `sibyl`",
+  "Use `sibyl` CLI for searching google/fetching webpages.",
+  "",
+  "## Usage",
+  "```",
+  "Commands:",
+  "  search <query>   Search the web",
+  "  fetch <url>      Fetch the content of a URL",
+  "  help             Show this help",
+  "  version          Show version",
+  "",
+  "Examples:",
+  '  sibyl search "react vite bootstrap"',
+  "  sibyl fetch https://vite.dev/guide",
+  "```",
+  "",
+  "Always search the web first when installing a new library/framework to get the latest docs. Also search when you want to find out information/API DOC etc.",
+  "Search more often than not. Conduct multiple search and fetches using subagents (with Haiku) and then proceed.",
+  "Use your judgment when interpreting web search results. Do not rely only on the first result you find.",
+  "If the information is incomplete, unclear, or insufficient to make a confident decision, perform a few more searches or fetch additional sources before making changes.",
+  "",
+  "Collect all the necessary information thoroughly before proceeding with creating/editing code.",
+].join("\n");
+
+// Filename for the standalone doc that reference-based targets (Claude, opencode) point at.
+export const SIBYL_DOC_FILENAME = "SIBYL.md";
+
+// Import line added to Claude Code's CLAUDE.md (resolves relative to that file).
+export const SIBYL_IMPORT_LINE = "@SIBYL.md";
+
+// Sentinel markers wrapping the embedded content in tools that can't reference a file.
+export const SIBYL_BLOCK_START = "<!-- SIBYL:START -->";
+export const SIBYL_BLOCK_END = "<!-- SIBYL:END -->";
+export const SIBYL_BLOCK_NOTICE =
+  "<!-- Managed by 'sibyl setup'. Do not edit or remove the block above; it is refreshed on each run. -->";

src/plugins/builtin-crawl4ai-fetch/main.test.ts

@@ -61,6 +61,9 @@ beforeEach(() => {
   parseFn.mockClear();
   envSnapshot = { ...process.env };
   delete process.env.SIBYL_CRAWL4AI_URL;
+  delete process.env.SIBYL_CRAWL4AI_PROXY_SERVER;
+  delete process.env.SIBYL_CRAWL4AI_PROXY_USERNAME;
+  delete process.env.SIBYL_CRAWL4AI_PROXY_PASSWORD;
 });
 
 afterEach(() => {
@@ -194,4 +197,47 @@ describe("builtin-crawl4ai-fetch", () => {
       expect.objectContaining({ method: "POST" }),
     );
   });
+
+  it("omits `proxy_config` when `SIBYL_CRAWL4AI_PROXY_SERVER` is unset", async () => {
+    const fetchMock = stubFetch(makeResponse({ json: { success: true, results: [] } }));
+
+    await fetchFn(url, context);
+
+    const mockCallArgs = fetchMock.mock.calls[0] as RequestInit[];
+    const req = mockCallArgs[1];
+    expect(req?.body).not.toContain("proxy_config");
+  });
+
+  it("includes `proxy_config` with server, username and password when all are set", async () => {
+    process.env.SIBYL_CRAWL4AI_PROXY_SERVER = "https://proxy.example.com:823";
+    process.env.SIBYL_CRAWL4AI_PROXY_USERNAME = "user";
+    process.env.SIBYL_CRAWL4AI_PROXY_PASSWORD = "pass";
+    const fetchMock = stubFetch(makeResponse({ json: { success: true, results: [] } }));
+
+    await fetchFn(url, context);
+
+    expect(fetchMock).toHaveBeenCalledWith(
+      expect.any(String),
+      expect.objectContaining({
+        body: expect.stringContaining(
+          '"proxy_config":{"server":"https://proxy.example.com:823","username":"user","password":"pass"}',
+        ),
+      }),
+    );
+  });
+
+  it("includes `proxy_config` with only server when username and password are unset", async () => {
+    process.env.SIBYL_CRAWL4AI_PROXY_SERVER = "https://proxy.example.com:823";
+    const fetchMock = stubFetch(makeResponse({ json: { success: true, results: [] } }));
+
+    await fetchFn(url, context);
+
+    const mockCallArgs = fetchMock.mock.calls[0] as RequestInit[];
+    const req = mockCallArgs[1];
+    const body = req?.body;
+
+    expect(body).toContain('"proxy_config":{"server":"https://proxy.example.com:823"}');
+    expect(body).not.toContain("username");
+    expect(body).not.toContain("password");
+  });
 });