feat: local MinerU mode via synchronous /file_parse

[gdccyuen]

What

Adds first-class support for parsing PDFs against a local (self-hosted) MinerU instance — the synchronous /file_parse endpoint — as an alternative to the cloud batch APIs. Self-hosted deployments that run MinerU on their own network (or fully offline) currently cannot use /file-urls/batch, /extract/task/batch, or /extract-results/batch (they 404 against local MinerU). This PR closes that gap.

Why

Design rationale lives in ADR 0001 of knowhere-self-hosted. Until this PR lands, self-hosted deployments have to ship a forked Docker image that patches pdf_service.py at build time. Once this merges and a release ships, that stopgap can be retired.

Changes

Config (packages/shared-python/shared/core/config/mineru.py)

Four new typed MineruConfig fields:

Field	Default	Purpose
MINERU_LOCAL_MODE	false	Gate. When true, parse_via_full dispatches to parse_via_local before any cloud-mode logic.
MINERU_LOCAL_LANG_LIST	"ch"	Language code for /file_parse's lang_list param. Local MinerU does not accept "auto"; "ch" covers Chinese, English, Japanese, Traditional Chinese, and Latin.
MINERU_LOCAL_BACKEND	"pipeline"	Backend (pipeline for CPU, vlm-engine for GPU).
MINERU_LOCAL_TIMEOUT	3600	Per-shard timeout in seconds for the sync /file_parse call. Includes queue wait when MINERU_SHARD_CONCURRENCY > 1.

Implementation (apps/worker/app/services/document_parser/providers/mineru/pdf_service.py)

• parse_via_local — POSTs the PDF as multipart form data to {MINERU_URL}/file_parse with lang_list, backend, and return_images=true. Streams remote URLs through a temp file first. Receives a ZIP, extracts it, flattens it.
• _flatten_extracted_zip — Local MinerU's ZIP has a nested {stem}/auto/{stem}.md + {stem}/auto/images/* layout. Downstream code expects full.md and an images/ directory at the output root. This lifts {stem}/auto/'s kept-ext contents to the output root (preserving the images/ subdir), excludes content_list / middle.json / model.json (parity with cloud's download_and_extract_zip exclude list), and renames the single .md to full.md. Hard-fails on zero or multiple .md files, and on zero or multiple {stem}/auto/ dirs, so we never silently pick the wrong one.
• _get_local_mineru_session — separate requests.Session with Retry(read=0, ...). Rationale: local MinerU is single-concurrency (max_concurrent_requests=1) by default; a ReadTimeout from queue wait must not cascade into a urllib3 retry that pushes the request to the back of the same queue. Cloud's build_mineru_session retries on [429, 502, 503, 504]; local mode drops 429 from status_forcelist and surfaces it as UnavailableException directly (no cloud quota manager — local has no API key).
• parse_via_full — top-of-function dispatch: if settings.MINERU_LOCAL_MODE: parse_via_local(...); return. Existing cloud flow untouched when MINERU_LOCAL_MODE=false.

Tests (apps/worker/tests/unit/test_pdf_service_flatten.py)

7 unit tests for _flatten_extracted_zip against synthetic ZIPs constructed in-memory:

• happy path: single {stem}/auto/{stem}.md + images/* → full.md + images/*
• error: no {stem}/auto/ dir (layout drift)
• error: multiple {stem}/auto/ dirs
• error: zero .md files
• error: multiple .md files
• preserved images even when no markdown is present (still raises on no-md, but images are lifted first)
• metadata JSON exclusion (content_list, middle.json, model.json dropped; keep.json retained)

Environment variables

Documented in self-hosted docs/configuration.md already (added in commit history of knowhere-self-hosted). Once merged here, the os.environ.get reads used by the stopgap patch in the self-hosted repo become proper typed pydantic_settings fields — the original ADR's "Future work" called this out explicitly.

Verification

uv run --directory apps/worker pytest tests/unit/test_pdf_service_flatten.py -q
7 passed in 0.32s

uv run --directory apps/worker pytest tests/unit -q
16 passed in 5.78s  # 9 existing + 7 new, no regressions

Smoke-test fixtures for actual end-to-end local MinerU testing live in knowhere-self-hosted's smoke test (which already covers local mode once this is shipped in a release).

Backward compatibility

MINERU_LOCAL_MODE defaults to false. Existing cloud deployments are entirely unaffected. Self-hosted deployments that want local mode set three env vars:

MINERU_LOCAL_MODE=true
MINERU_URL=http://host.docker.internal:8000
# MINERU_API_KEYS may be left as placeholder; local mode never reads it

Out of scope (future PRs)

• Async /tasks API for very large PDFs (sync /file_parse timeout becomes the bottleneck on multi-shard large PDFs).
• VLM backend smoke test (requires GPU CI).