refactor!: prepare pipeline foundations for deterministic cleanup#2
Merged
Conversation
…rnative to Firecrawl - New built-in SourceProvider over docling-serve POST /v1/convert/source - Single sources+kind request shape (verified stable from 1.0.0 through latest) - X-Api-Key header support, image_export_mode hardcoded to placeholder - Config: baseUrl (required), apiKey, doOcr (default true), tableMode (fast/accurate) - Registered as default-docling in the default engine descriptor bundle - Both default-firecrawl and default-docling ship; SOURCE_PROVIDER env var selects active - Step renamed firecrawl -> fetch (provider-agnostic name) - Docs: CUSTOMIZATION.md, CONFIGURATION.md, ARCHITECTURE.md, ROADMAP.md updated - Configuration validation redesign noted in ROADMAP short-term section
…nfig surface, guard non-text content Redesign Firecrawl and Docling provider config surface: - Replace formats/maxAge with output knob (markdown|html|rawHtml) - Add stripBase64Images, parsePdf knobs (Firecrawl) - Add output knob (Docling) Introduce new HTTP source provider (default-http): - Testing-only fallback: direct fetch, no SSRF protection, no external service - Config knobs: userAgent, maxBytes, titleFromHtml - Streaming body read with configurable maxBytes cap (memory-safe) - Non-text responses are rejected with unsupported_media_type (Phase 1): * Content-Type allowlist (text/*, application/json|xml|xhtml+xml, +json/+xml) * NUL-byte sniff as fallback for missing/mislabeled headers SourceDocument gains truncated flag; LoadSourceStep degrades on truncation Document value semantics: reported sources carry output type in CUSTOMIZATION.md Add REFACTORING.md with staged content-typed body plan (media type + binary)
…d preserve parsed types
Introduce text and binary body representations and propagate mediaType across source providers, steps, renderers, and reports. Return binary bodies from the HTTP provider, fail terminal binary runs with unconverted_binary, and skip text-only steps on unsupported media types. Rename representation-spanning diagnostics from chars to length and omit ratios across representation changes.
Implements the three pillars of the REFACTORING design: - Lazy registries: providers and output renderers are built and validated only when an active pipeline references them (D1). - Pipeline activation: tri-state enabled on pipelines (true/false/ omitted=referenced) resolved in the app layer (D2-D3). - Dumb env substitution: missing VAR with no :- default resolves to empty string instead of throwing, so the env layer never fails for unused config. Required-ness is owned by reachable built-in schemas (D4). Changes by area: Engine: Tighten provider/renderer create to synchronous (D5). Ship activation-lazy and activation-pipeline tests (D1-D3,D7). Add built-names tracking for skipped-leaf startup log (D7). Config: yaml-config: add optional enabled field to pipeline schema. yaml-app-config: resolve tri-state via adapter references. env-substitution: remove missing_env throw; unset var -> "". Built-in config schemas: field-named error messages (D6). Pipelines: Rename default -> clean-llm (Firecrawl + LLM). Add truncate pipeline (load-source + truncate, no LLM). Adapt YAML, compose files, and README for DEFAULT_PIPELINE. App: Add --check mode (compose without listen) (D8). Compose: Remove :? guards from FIRECRAWL_BASE_URL, DOCLING_BASE_URL, LLM_BASE_URL, LLM_MODEL. Add DEFAULT_PIPELINE forwarding. Docs: Update ARCHITECTURE, CONFIGURATION, CUSTOMIZATION, ROADMAP, smoke-testing, README, .env.example for lazy validation, pipeline activation, and new defaults.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for deterministic cleanup: rework the source providers, body
model, and configuration so non-LLM steps can be composed and validated
cleanly. Adds the first no-LLM pipeline (truncate).
Source providers
stripBase64Images and parsePdf for Firecrawl.
Content-typed body (breaking)
unsupported media; terminal binary output fails with unconverted_binary.
omit ratios across representation changes.
Configuration (breaking)
renderers are built and validated only when an active pipeline references them.
Docs and scripts updated to match.