refactor!: prepare pipeline foundations for deterministic cleanup

.env.example

@@ -22,6 +22,9 @@ LOG_LEVEL=info
 LOG_PRETTY=auto
 
 # --- Pipeline ------------------------------------------------------------
+# Active pipeline selected by HTTP adapters. Default: truncate (load-source + truncate)
+DEFAULT_PIPELINE=truncate
+
 # Maximum concurrent source-loading groups. Default: 1
 SOURCE_CONCURRENCY=4
 
@@ -38,20 +41,29 @@ DEBUG_XML_INCLUDE_SKIPPED=false
 OUTPUT_TARGET_CHARS=35000
 
 # --- Source provider -----------------------------------------------------
-# Firecrawl base URL used by the default source provider. Required, no default.
+# Active source provider name used by pipelines (truncate and clean-llm). Default: default-http (native HTTP fetch, no external service needed)
+SOURCE_PROVIDER=default-http
+
+# Firecrawl base URL. Required only when a referenced provider uses it (e.g. default-firecrawl).
 FIRECRAWL_BASE_URL=https://firecrawl.example
 
 # Optional Firecrawl bearer token. Default: empty (no token)
 FIRECRAWL_API_KEY=
 
+# Docling Serve base URL. Required only when a referenced provider uses it (e.g. default-docling).
+DOCLING_BASE_URL=https://docling.example
+
+# Optional Docling API key (X-Api-Key header). Default: empty (no token)
+DOCLING_API_KEY=
+
 # --- LLM provider --------------------------------------------------------
-# OpenAI-compatible Chat Completions /v1 base URL. Required, no default.
+# OpenAI-compatible Chat Completions /v1 base URL. Required only when a referenced provider uses it (e.g. default-llm).
 LLM_BASE_URL=https://openai-compatible.example/v1
 
 # Optional LLM bearer token. Default: empty (no token)
 LLM_API_KEY=
 
-# Model identifier sent to the chat-completions endpoint. Required, no default.
+# Model identifier sent to the chat-completions endpoint. Required only when a referenced provider uses it (e.g. default-llm).
 LLM_MODEL=model-name
 
 # Model context window in tokens. Default: empty (disables the context-fit gate)

AGENTS.md

@@ -22,6 +22,7 @@ Read the row that matches your task. Each focused doc is self-contained for its
 | Run or configure the service, or change the environment surface                        | [docs/CONFIGURATION.md](docs/CONFIGURATION.md), [README.md](README.md)                                               |
 | Work on logging, request context, errors, or diagnostics                               | [docs/agents/logging-and-errors.md](docs/agents/logging-and-errors.md)                                               |
 | Touch auth, URL validation, upstream parsing, XML diagnostics, or other security paths | [docs/agents/security-boundaries.md](docs/agents/security-boundaries.md)                                             |
+| Run smoke tests: start/stop, interpret footers, avoid common traps                     | [docs/agents/smoke-testing.md](docs/agents/smoke-testing.md)                                                         |
 | Change code style, comments, TypeScript conventions, or tests                          | [docs/agents/coding-conventions.md](docs/agents/coding-conventions.md), [docs/TESTING.md](docs/TESTING.md)           |
 | Commit, push, release, or change versioning and deployment                             | [docs/agents/git-workflow.md](docs/agents/git-workflow.md), [docs/RELEASING.md](docs/RELEASING.md)                   |
 | Check scope before adding a feature                                                    | [docs/ROADMAP.md](docs/ROADMAP.md)                                                                                   |
@@ -34,3 +35,5 @@ Read the row that matches your task. Each focused doc is self-contained for its
 - Do not edit `package.json`'s `version` field by hand. Version bumps use `npm version` only when explicitly requested.
 - When changing the configuration surface, update the source schema, [config/llm-context-loader.yaml](config/llm-context-loader.yaml), [.env.example](.env.example), Compose files when relevant, and the configuration and customization docs in the same change.
 - When changing behavior, keep the docs that describe it accurate in the same change.
+- No gratuitous comments. Inline comments that narrate a change are prohibited ([coding-conventions](docs/agents/coding-conventions.md)). Before adding a comment, ask: will it still be useful after the transient context is gone?
+- Don't over-document trivial changes. A new knob, helper, or internal refactor does not warrant updates to `ARCHITECTURE.md` or `extension-authoring.md`. Update only docs that directly describe the changed surface.

README.md

@@ -7,18 +7,14 @@
 [![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![node](https://img.shields.io/badge/node-22+-brightgreen.svg)](https://nodejs.org/)
 
-LLM Context Loader is a small HTTP service that turns URLs into markdown suitable for language-model context. The default bundle exposes an Open WebUI external web loader endpoint and a limited Jina Reader-style endpoint, fetches pages through Firecrawl, can run OpenAI-compatible Chat Completions passes, and renders either plain markdown or markdown with an XML diagnostic footer.
+LLM Context Loader is a small HTTP service that turns URLs into markdown for LLM context. It exposes Open WebUI and Jina Reader-style HTTP endpoints, fetches pages through built-in or external providers, can run LLM passes, and renders markdown with an optional XML diagnostic footer. It handles single URLs — crawling, search, model serving, storage, and retrieval stay outside this tool.
 
 ## Current Shape
 
-- HTTP adapters: Open WebUI `POST /`, Jina-style `GET /r/<url>` and `GET /r?url=<url>`, plus open `GET /health`.
-- Source provider: Firecrawl `/v2/scrape` with markdown output.
-- LLM provider: lowest-common-denominator OpenAI-compatible `/chat/completions` using `system` and `user` messages.
-- Default pipeline: load the page, clean it with an LLM pass, summarize it when it is too long, cap the output size.
-- LLM passes are guarded by URL-hallucination detection that rolls back to trusted source content.
-- Output renderers: `debug-xml` and `passthrough`, selected per pipeline in YAML.
-
-It fetches and shapes single URLs; crawling, search, model serving, storage, and retrieval stay outside this tool.
+- **Pipelines:** `truncate` (default, no LLM) and `clean-llm` (Firecrawl + LLM), selected via `DEFAULT_PIPELINE`.
+- **Source providers:** native HTTP fetch, Firecrawl, Docling.
+- **LLM provider:** OpenAI-compatible `/chat/completions`.
+- **Output renderers:** `debug-xml` and `passthrough`, selected per pipeline in YAML.
 
 ## Quick Start
 
@@ -27,7 +23,6 @@ Requires Node 22+ for local runs.
 ```bash
 npm install
 cp .env.example .env
-# edit .env for FIRECRAWL_BASE_URL, LLM_BASE_URL, LLM_MODEL, and keys if needed
 npm run dev
 ```
 
@@ -68,7 +63,7 @@ docker compose -f compose.deploy.yaml up -d
 
 The deploy compose file requires `LLMC_IMAGE_TAG`. The example env file uses `latest`, but repeatable deployments should pin it to an immutable release tag.
 
-The provider values in `.env.example` are placeholders. Set `FIRECRAWL_BASE_URL`, `LLM_BASE_URL`, and `LLM_MODEL` before starting the service; the Compose files fail early if they are missing. If Firecrawl or the LLM server runs on the host machine, remember that `localhost` inside a container means the container itself. Use a LAN address or `host.docker.internal` when appropriate.
+The out-of-box `truncate` pipeline needs no provider config. Switch to `clean-llm` with `DEFAULT_PIPELINE=clean-llm` and the appropriate provider vars. The Compose files pass all variables through without failing early; the service validates only the active pipeline's providers at startup. If Firecrawl or the LLM server runs on the host, use a LAN address or `host.docker.internal` instead of `localhost`.
 
 ## API
 

compose.deploy.yaml

@@ -10,6 +10,9 @@ x-llmc-environment: &llmc-environment # Bootstrap environment read before YAML l
   LOG_LEVEL: "${LOG_LEVEL:-info}"
   LOG_PRETTY: "${LOG_PRETTY:-auto}"
 
+  # Active pipeline selected by HTTP adapters.
+  DEFAULT_PIPELINE: "${DEFAULT_PIPELINE:-truncate}"
+
   # Pipeline concurrency and renderer selection.
   SOURCE_CONCURRENCY: "${SOURCE_CONCURRENCY:-1}"
   LLM_CONCURRENCY: "${LLM_CONCURRENCY:-1}"
@@ -18,13 +21,16 @@ x-llmc-environment: &llmc-environment # Bootstrap environment read before YAML l
   OUTPUT_TARGET_CHARS: "${OUTPUT_TARGET_CHARS:-25000}"
 
   # Source provider.
-  FIRECRAWL_BASE_URL: "${FIRECRAWL_BASE_URL:?Set FIRECRAWL_BASE_URL in .env or the shell}"
+  SOURCE_PROVIDER: "${SOURCE_PROVIDER:-default-http}"
+  FIRECRAWL_BASE_URL: "${FIRECRAWL_BASE_URL:-}"
   FIRECRAWL_API_KEY: "${FIRECRAWL_API_KEY:-}"
+  DOCLING_BASE_URL: "${DOCLING_BASE_URL:-}"
+  DOCLING_API_KEY: "${DOCLING_API_KEY:-}"
 
   # LLM provider.
-  LLM_BASE_URL: "${LLM_BASE_URL:?Set LLM_BASE_URL in .env or the shell}"
+  LLM_BASE_URL: "${LLM_BASE_URL:-}"
   LLM_API_KEY: "${LLM_API_KEY:-}"
-  LLM_MODEL: "${LLM_MODEL:?Set LLM_MODEL in .env or the shell}"
+  LLM_MODEL: "${LLM_MODEL:-}"
   LLM_CONTEXT_TOKENS: "${LLM_CONTEXT_TOKENS:-}"
   LLM_CHARS_PER_TOKEN: "${LLM_CHARS_PER_TOKEN:-3.5}"
   LLM_SAFETY_MARGIN_TOKENS: "${LLM_SAFETY_MARGIN_TOKENS:-128}"

compose.yaml

@@ -10,6 +10,9 @@ x-llmc-environment: &llmc-environment # Bootstrap environment read before YAML l
   LOG_LEVEL: "${LOG_LEVEL:-info}"
   LOG_PRETTY: "${LOG_PRETTY:-auto}"
 
+  # Active pipeline selected by HTTP adapters.
+  DEFAULT_PIPELINE: "${DEFAULT_PIPELINE:-truncate}"
+
   # Pipeline concurrency and renderer selection.
   SOURCE_CONCURRENCY: "${SOURCE_CONCURRENCY:-1}"
   LLM_CONCURRENCY: "${LLM_CONCURRENCY:-1}"
@@ -18,13 +21,16 @@ x-llmc-environment: &llmc-environment # Bootstrap environment read before YAML l
   OUTPUT_TARGET_CHARS: "${OUTPUT_TARGET_CHARS:-25000}"
 
   # Source provider.
-  FIRECRAWL_BASE_URL: "${FIRECRAWL_BASE_URL:?Set FIRECRAWL_BASE_URL in .env or the shell}"
+  SOURCE_PROVIDER: "${SOURCE_PROVIDER:-default-http}"
+  FIRECRAWL_BASE_URL: "${FIRECRAWL_BASE_URL:-}"
   FIRECRAWL_API_KEY: "${FIRECRAWL_API_KEY:-}"
+  DOCLING_BASE_URL: "${DOCLING_BASE_URL:-}"
+  DOCLING_API_KEY: "${DOCLING_API_KEY:-}"
 
   # LLM provider.
-  LLM_BASE_URL: "${LLM_BASE_URL:?Set LLM_BASE_URL in .env or the shell}"
+  LLM_BASE_URL: "${LLM_BASE_URL:-}"
   LLM_API_KEY: "${LLM_API_KEY:-}"
-  LLM_MODEL: "${LLM_MODEL:?Set LLM_MODEL in .env or the shell}"
+  LLM_MODEL: "${LLM_MODEL:-}"
   LLM_CONTEXT_TOKENS: "${LLM_CONTEXT_TOKENS:-}"
   LLM_CHARS_PER_TOKEN: "${LLM_CHARS_PER_TOKEN:-3.5}"
   LLM_SAFETY_MARGIN_TOKENS: "${LLM_SAFETY_MARGIN_TOKENS:-128}"

config/llm-context-loader.yaml

@@ -1,16 +1,16 @@
 schemaVersion: 1
 
-# HTTP adapters expose the default pipeline through client-specific contracts.
+# HTTP adapters expose the selected pipeline through client-specific contracts.
 httpAdapters:
   open-webui:
     type: open-webui
-    pipeline: default
+    pipeline: ${DEFAULT_PIPELINE:-truncate}
     config:
       auth:
         bearerToken: ${OWUI_AUTH_TOKEN:-}
   jina:
     type: jina
-    pipeline: default
+    pipeline: ${DEFAULT_PIPELINE:-truncate}
     config:
       auth:
         bearerToken: ${JINA_AUTH_TOKEN:-}
@@ -28,50 +28,78 @@ outputRenderers:
 
 # Source providers turn an input URL into source markdown.
 sourceProviders:
+  default-http:
+    type: http
+    config:
+      userAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
+      maxBytes: 5000000
+      titleFromHtml: true
   default-firecrawl:
     type: firecrawl
     config:
-      baseUrl: ${FIRECRAWL_BASE_URL}
+      baseUrl: ${FIRECRAWL_BASE_URL:-}
       apiKey: ${FIRECRAWL_API_KEY:-}
+      output: markdown
       onlyMainContent: true
-      formats: [markdown]
-      maxAge: 0
+      stripBase64Images: true
+      parsePdf: true
+  default-docling:
+    type: docling
+    config:
+      baseUrl: ${DOCLING_BASE_URL:-}
+      apiKey: ${DOCLING_API_KEY:-}
+      output: markdown
+      doOcr: true
+      tableMode: accurate
 
 # LLM providers run prompt-rendered chat-completions passes.
 llmProviders:
   default-llm:
     type: openai-chat
     config:
-      baseUrl: ${LLM_BASE_URL}
+      baseUrl: ${LLM_BASE_URL:-}
       apiKey: ${LLM_API_KEY:-}
-      model: ${LLM_MODEL}
+      model: ${LLM_MODEL:-}
       contextTokens: ${LLM_CONTEXT_TOKENS:-}
       charsPerToken: ${LLM_CHARS_PER_TOKEN:-3.5}
       safetyMarginTokens: ${LLM_SAFETY_MARGIN_TOKENS:-128}
       extraBody: {}
 
-# The default pipeline fetches, cleans, verifies URLs, summarizes, and caps output size.
+# Shipped pipelines. DEFAULT_PIPELINE selects which is active (truncate by default).
 pipelines:
-  default:
+  truncate:
+    outputRenderer: ${DEFAULT_OUTPUT_RENDERER:-debug-xml}
+    limiters:
+      source: ${SOURCE_CONCURRENCY:-1}
+    steps:
+      - type: load-source
+        name: fetch
+        concurrencyGroup: source
+        timeoutSeconds: ${LOAD_SOURCE_TIMEOUT_SECONDS:-20}
+        config:
+          provider: ${SOURCE_PROVIDER:-default-http}
+      - type: truncate
+        name: truncate
+        config:
+          targetChars: ${OUTPUT_TARGET_CHARS:-25000}
+
+  clean-llm:
     outputRenderer: ${DEFAULT_OUTPUT_RENDERER:-debug-xml}
     limiters:
       source: ${SOURCE_CONCURRENCY:-1}
       llm: ${LLM_CONCURRENCY:-1}
     steps:
-      # Fetch source markdown and keep source work in the source limiter group.
       - type: load-source
-        name: firecrawl
+        name: fetch
         concurrencyGroup: source
         timeoutSeconds: ${LOAD_SOURCE_TIMEOUT_SECONDS:-20}
         config:
-          provider: default-firecrawl
-      # Capture trusted source URLs before any LLM pass can alter the body.
+          provider: ${SOURCE_PROVIDER:-default-firecrawl}
       - type: capture-urls
         name: capture_source_urls
         concurrencyGroup: source
         config:
           artifact: trusted-urls
-      # Clean readable source markdown while preserving roughly the original budget.
       - type: llm-pass
         name: clean
         concurrencyGroup: llm
@@ -83,15 +111,13 @@ pipelines:
           templates:
             system: ../templates/clean.system.md
             user: ../templates/clean.user.md
-      # Roll back to source content when the clean pass introduces new URLs.
       - type: verify-urls
         name: verify_after_clean
         concurrencyGroup: llm
         config:
           artifact: trusted-urls
           onHallucination: rollback
           maxReportedUrls: 0
-      # Summarize only once content exceeds the target output size.
       - type: llm-pass
         name: summarize
         concurrencyGroup: llm
@@ -105,15 +131,13 @@ pipelines:
             user: ../templates/summarize.user.md
             vars:
               targetChars: ${OUTPUT_TARGET_CHARS:-25000}
-      # Report summary URL drift without replacing the current body.
       - type: verify-urls
         name: verify_after_summarize
         concurrencyGroup: llm
         config:
           artifact: trusted-urls
           onHallucination: report
           maxReportedUrls: 50
-      # Enforce the final hard character budget.
       - type: truncate
         name: truncate
         config:

docs/ARCHITECTURE.md

@@ -66,7 +66,7 @@ scripts/                     developer smoke helpers
 The current built-ins are:
 
 - HTTP adapters: `open-webui`, `jina`.
-- Source providers: `firecrawl`.
+- Source providers: `http`, `firecrawl`, `docling`.
 - LLM providers: `openai-chat`.
 - Pipeline steps: `load-source`, `llm-pass`, `truncate`, `capture-urls`, `verify-urls`.
 - Output renderers: `debug-xml`, `passthrough`.
@@ -99,6 +99,15 @@ One important boundary is explicit in the architecture test: `src/core/` must no
 
 `src/core/` is the framework-free runtime engine. It does not know about Firecrawl, OpenAI, Open WebUI, Jina, or provider categories.
 
+The pipeline body is a **discriminated union** (`TextBody | BinaryBody`), discriminated on `kind`:
+
+- **Text bodies** (`kind: "text"`) carry `content` as a `string` and are the payload text-only steps read and write.
+- **Binary bodies** (`kind: "binary"`) carry `bytes` as a `Uint8Array` and are carried through the pipeline for conversion by a later step.
+
+A binary body that reaches the end of the pipeline without being converted is treated as a **failed run** with the error `unconverted_binary: <mediaType>`.
+
+Run reports use representation-neutral length diagnostics: `initial_length` and `final_length` are characters for text bodies and bytes for binary bodies. `ratio` is reported only when the first and final body have the same representation.
+
 Key pieces:
 
 - `PipelineStep` (`src/contracts/pipeline/step.ts`) is the executable step port: `run(ctx): Promise<StepResult>`.
@@ -158,10 +167,12 @@ Bootstrap environment is intentionally small and parsed by `src/config/env-confi
 The YAML document is translated to `AppConfig` before runtime construction:
 
 1. `src/config/yaml/yaml-config.ts` validates the coarse YAML shape: `httpAdapters`, `outputRenderers`, `sourceProviders`, `llmProviders`, and `pipelines`.
-2. `src/config/yaml/yaml-app-config.ts` maps engine-owned sections to `AppConfig.engineConfig`, HTTP adapter declarations to `AppConfig.adapters.http`, and `schemaVersion` to app metadata.
+2. `src/config/yaml/yaml-app-config.ts` maps engine-owned sections to `AppConfig.engineConfig`, HTTP adapter declarations to `AppConfig.adapters.http`, and `schemaVersion` to app metadata. Pipeline activation follows a tri-state `enabled`: `true` compiles, `false` parks it, **omitted** activates only when an HTTP adapter references it.
 3. Each built-in descriptor parses its own `config` block with a local schema during engine or adapter construction.
 
-YAML environment substitution happens before schema validation. See [CONFIGURATION.md](CONFIGURATION.md) for default operation and [CUSTOMIZATION.md](CUSTOMIZATION.md) for YAML structure.
+YAML environment substitution happens before schema validation. A missing `${VAR}` with no `:-` default resolves to `""` so the env layer never throws for unused config. Required-ness is owned by reachable built-in schemas, not by substitution.
+
+**Lazy validation**: providers and output renderers are built and validated only when an active pipeline references them. An unused provider with missing env vars does not fail startup.
 
 `EngineConfig` excludes adapter declarations and schema metadata. `AppConfig` is the app-level envelope that pairs `engineConfig` with adapter configuration such as `adapters.http`.