From d76ed8aaa821ead21a9531c5efcb6d92555aaeb6 Mon Sep 17 00:00:00 2001 From: Kresna Date: Thu, 30 Jul 2026 16:12:10 +0700 Subject: [PATCH 01/10] =?UTF-8?q?docs(spec):=20Image=20to=20Text=20(OCR)?= =?UTF-8?q?=20design=20=E2=80=94=20English=20MVP?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PP-OCR pipeline on ONNX Runtime Web (WebGPU/WASM), single image + PDF input, review-&-adjust preprocessing, reason-specific errors. Structured receipt parsing, multi-language, and batch deferred to later phases. --- .../specs/2026-07-30-image-ocr-design.md | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-30-image-ocr-design.md diff --git a/docs/superpowers/specs/2026-07-30-image-ocr-design.md b/docs/superpowers/specs/2026-07-30-image-ocr-design.md new file mode 100644 index 0000000..7cbda07 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-image-ocr-design.md @@ -0,0 +1,128 @@ +# Image → Text (OCR) — Design Spec + +**Status:** Approved (brainstorming complete) +**Date:** 2026-07-30 +**Author:** brainstormed with Kresna +**Scope:** Phase 1 (English-only MVP). Structured receipt parsing, multi-language, and batch input are explicitly deferred to later phases. + +## Goal + +Add a client-side **Image → Text (OCR)** tool to GoodWebTools that extracts raw text from an uploaded image or PDF entirely on-device. Receipts are the motivating use case, but the tool is a general OCR utility. The uploaded document never leaves the browser; only the OCR model files are fetched (from a CDN, cached thereafter). + +## Motivation & Context + +Users want to pull text out of receipts, screenshots, and scanned documents without uploading them to a third-party service. GWT already ships heavy on-device WASM/ML tools (`onnxruntime-web`, `mupdf`, `upscaler`, `@tensorflow/tfjs`), so the infrastructure patterns for lazy-loaded WASM, worker inference, and PWA-precache exclusion already exist and will be reused. + +## Locked Decisions + +These were settled during brainstorming and are not open questions: + +1. **Output:** raw recognized text (copy + download `.txt`). Structured field extraction (merchant/date/total/line items) is a **future phase**, not this spec. +2. **Language:** **English only** for the MVP. The architecture must leave a clean seam for a curated (~10-language) picker later. +3. **Engine:** the **PP-OCR pipeline (detection + angle-classification + recognition) running on ONNX Runtime Web**, via a maintained browser SDK (see Engine section). Chosen over Tesseract.js for higher accuracy on messy/real-world images ("the most powerful one"). +4. **Input:** a **single image** (PNG/JPG/WebP, via upload / drag-drop / paste) **or a PDF** (rasterized via the existing mupdf pipeline; multi-page PDFs get page selection). +5. **Preprocessing:** a **"review & adjust" step** before OCR. Light auto-cleanup is available but **off by default** (PP-OCR is trained on natural images; aggressive binarizing can *reduce* accuracy). Manual overrides: rotate, crop, threshold slider. +6. **Model hosting:** language/model files are **fetched from a CDN on demand** and cached. The image itself never leaves the browser; the UI discloses the one-time model download. +7. **Error messages:** every failure/degradation surfaces its **specific reason** (see Error Handling). + +## Engine + +The OCR engine is a PP-OCR (PaddleOCR) pipeline running on **ONNX Runtime Web** (already a project dependency), with **WebGPU acceleration and automatic WASM fallback**. The full pipeline — image normalization, text **detection**, per-box crop/rectify, angle **classification**, text **recognition**, and CTC **decoding** — plus all post-processing is provided by the SDK; we do **not** hand-roll contour finding / polygon-unclip / perspective warp. + +**Lead candidate:** [`ppu-paddle-ocr`](https://www.npmjs.com/package/ppu-paddle-ocr) (`/web` entry) — PP-OCRv5, ONNX Runtime Web, WebGPU→WASM auto-fallback, on-demand + cached model download, `initialize()/recognize()/destroy()` API. +**Alternatives with the same shape:** official [`@paddleocr/paddleocr-js`](https://www.npmjs.com/package/@paddleocr/paddleocr-js), [`@gutenye/ocr`](https://github.com/gutenye/ocr). + +**Engine-agnostic design:** all candidates share the `init → recognize(canvas) → { text, boxes }` shape. The exact package + version is pinned in the **first implementation task** via a thin vertical-slice eval (recognize text on one canvas, confirm bundle/model sizes, WebGPU + WASM paths). If the lead candidate fails the eval, an alternative is substituted behind the same `ocr.lib` wrapper with no other code changes. + +## Architecture + +A new **Image**-category tool `image-ocr` ("Image to Text (OCR)"), lazy-loaded through the registry `load()` like other heavy tools. Business logic lives in small, isolated, unit-tested libs; the island is a thin orchestrator. + +``` +File/paste ─┐ +PDF ────────┤→ [source → canvas]* → [Review & Adjust] → [ocr.lib.recognize] → [Results] + │ *PDF rasterized preprocess SDK on ORT-web text + copy + │ via mupdf (optional) WebGPU/WASM + download +``` + +### Components + +| File | Responsibility | Depends on | Tested by | +|------|----------------|------------|-----------| +| `src/tools/image/ocr.lib.ts` | Sole contact point with the OCR SDK. Lazy cached `initEngine()`; `recognize(source) → { text, lines: [{ text, box, confidence }] }`; maps SDK/runtime failures to reason-specific `OcrError`s. | the OCR SDK (mocked in tests) | mock SDK: output mapping + each error path | +| `src/tools/image/ocr-preprocess.lib.ts` | Pure canvas transforms for the review step: grayscale, contrast/threshold (reuse `mono.lib` `toGrayscale`/`toBlackWhite`), rotate, crop. Input canvas → output canvas. | `mono.lib`, `canvas.lib` | deterministic pixel assertions (mirrors `mono.lib.test`) | +| `src/tools/image/ocr-pdf.lib.ts` | Thin adapter: PDF `File` + page index → canvas, using the same mupdf rasterization the `pdf-to-image` tool uses. Exposes page count for the picker. | existing mupdf client/worker (mocked) | mock mupdf: page count + rasterize call | +| `src/islands/image/ImageOcr.tsx` | Thin UI orchestrator: input (dropzone + `usePasteImage`), PDF page selection, review-&-adjust panel, run + progress, results (editable text, copy, download, optional overlay). | the three libs above | light island test / manual smoke | +| `src/registry/tools.ts` (edit) | Register `id: 'image-ocr'`, `category: 'Image'`, `route: '/tools/image-ocr'`, `status: 'beta'`, `keywords: ['ocr','receipt','scan','text','extract','recognize','read']`, `load: () => import('@/islands/image/ImageOcr')`. | — | build / registry render | +| `astro.config.mjs` (edit) | Add the ORT-wasm + OCR SDK chunk globs to workbox `globIgnores` so they are not PWA-precached (same treatment as DbDiagram/Monaco). | — | build (precache manifest) | +| route page `src/pages/tools/image-ocr.astro` | Standard tool route wrapper hosting the island (follow existing tool pages). | — | build | + +### Data Model + +```ts +interface OcrLine { + text: string; + box: { x: number; y: number; width: number; height: number }; // for optional overlay + confidence: number; // 0..1 +} +interface OcrResult { + text: string; // all lines joined in reading order (sorted top-to-bottom, then left-to-right) + lines: OcrLine[]; +} +class OcrError extends Error { + reason: 'engine-unsupported' | 'model-download' | 'inference' | 'no-text' | 'input'; + // message is the user-facing, reason-specific string +} +``` + +## UX / Data Flow + +1. **Input:** drag-drop / file-pick / paste (`usePasteImage`) an image, or pick a PDF. A PDF renders a page to canvas via `ocr-pdf.lib`; multi-page PDFs show a page selector. +2. **Review & Adjust panel:** live canvas preview with controls — + - **"Clean up image" toggle** (grayscale + contrast/threshold), **off by default**. + - **Rotate** (90° steps + fine). + - **Crop** to the region of interest. + - **Threshold slider** (only meaningful when cleanup is on). +3. **Run OCR:** button runs `ocr.lib.recognize()` on the adjusted canvas. A progress/indeterminate indicator shows while the engine initializes (first-run model download) and infers. Disclose on first run that models download once from a CDN and are then cached. +4. **Results:** recognized text in an **editable/selectable textarea**, with **Copy** and **Download .txt**. Empty result → "no text detected" guidance. +5. **Optional stretch (flagged, not MVP-blocking):** a bounding-box overlay on the preview using `lines[].box`. + +## Error Handling (reason-specific) + +A `reason → user message` map lives in `ocr.lib`. Every surfaced state states *why*: + +- **WebGPU unavailable** → informational, not an error: *"Running in slower CPU (WASM) mode — WebGPU isn't available in this browser."* (OCR still works.) +- **Engine can't initialize at all** (e.g. no WASM SIMD) → *"On-device OCR couldn't start: this browser lacks WebAssembly SIMD, which the OCR engine requires."* (`reason: 'engine-unsupported'`) +- **Model download fails** (offline / CDN blocked) → *"Couldn't download the OCR model (network error or blocked). First use needs a connection — check it and Retry."* + Retry button. (`reason: 'model-download'`) +- **Inference throws** → surface the underlying reason text: *"OCR failed: <engine message>."* (`reason: 'inference'`) +- **Oversized image** → auto-downscaled to a max dimension before inference; note *"Large image downscaled to N px for processing."* (guard, not a hard error) +- **No text found** → *"No text detected — try turning on Clean up image, or use a clearer/tighter crop."* (`reason: 'no-text'`) +- **Unsupported file / broken PDF** → validation message naming the problem. (`reason: 'input'`) + +## Testing + +- **Unit (Vitest, following existing patterns):** + - `ocr-preprocess.lib` — deterministic pixel assertions for grayscale/threshold/rotate/crop (mirrors `mono.lib.test`). + - `ocr.lib` — mock the SDK; assert SDK output → `OcrResult` mapping, reading-order join, and each `OcrError` reason/message. + - `ocr-pdf.lib` — mock mupdf; assert page-count read and rasterize invocation. +- **Not unit-tested:** real model inference (large, nondeterministic) — covered by a **manual smoke checklist** (English receipt, screenshot, multi-page PDF, WebGPU path, WASM-fallback path, offline model-download error). This mirrors how capture/hotkey/WASM services mock their heavy dependency in unit tests. +- Keep the island thin so coverage concentrates in the libs. + +## Constraints & Non-Functional + +- **Privacy:** the document never leaves the browser. Only model files are fetched (CDN, cached). UI discloses this. +- **Bundle/PWA:** ORT-wasm + SDK chunks excluded from workbox precache via `globIgnores`; models are runtime-fetched (optionally runtime-cached by the service worker for repeat/offline use). +- **Performance:** inference off the main thread (ORT-web workers); oversized inputs downscaled; progress surfaced. +- **Follows existing patterns:** registry `ToolDef` shape, `usePasteImage`, `mono.lib`/`canvas.lib`, mupdf rasterization, lazy `load()`. + +## Out of Scope (future phases) + +- Structured receipt fields (merchant, date, total, tax, line items). +- Multi-language picker (~10 curated languages + per-language rec models/dicts). +- Batch / multi-file OCR. +- Auto-deskew of the whole image (PP-OCR per-box rectification largely covers rotation). +- Self-hosting models for fully-offline first use. + +## Open Implementation Detail (resolved in the plan, not blocking) + +- Exact SDK package + version, and its model-hosting URLs, are pinned in the first implementation task's vertical-slice eval. The `ocr.lib` wrapper isolates this choice so a substitution touches one file. From 1cc7996e526d95bbb63df6f57fd3c28b431191cf Mon Sep 17 00:00:00 2001 From: Kresna Date: Thu, 30 Jul 2026 16:20:16 +0700 Subject: [PATCH 02/10] =?UTF-8?q?docs(plan):=20Image=20to=20Text=20(OCR)?= =?UTF-8?q?=20implementation=20plan=20=E2=80=94=204=20TDD=20tasks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../superpowers/plans/2026-07-30-image-ocr.md | 895 ++++++++++++++++++ 1 file changed, 895 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-30-image-ocr.md diff --git a/docs/superpowers/plans/2026-07-30-image-ocr.md b/docs/superpowers/plans/2026-07-30-image-ocr.md new file mode 100644 index 0000000..8ee5614 --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-image-ocr.md @@ -0,0 +1,895 @@ +# Image → Text (OCR) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a client-side "Image → Text (OCR)" tool that extracts raw text from an image or PDF page entirely on-device using a PP-OCR pipeline on ONNX Runtime Web. + +**Architecture:** Three small, isolated, unit-tested libs (`ocr-preprocess.lib`, `ocr-pdf.lib`, `ocr.lib`) plus a thin orchestrator island (`ImageOcr.tsx`). The OCR SDK is quarantined to one tiny file (`ocr.engine.ts`) so the exact package is swappable behind a stable interface. The tool is registered in the existing registry and served by the existing dynamic `[tool].astro` route. + +**Tech Stack:** Astro + React island; `ppu-paddle-ocr` (PP-OCRv5 on ONNX Runtime Web, WebGPU→WASM); existing `mono.lib` (grayscale/threshold), `render.lib` (`pdfjs-dist` rasterization), `downloadService`, `usePasteImage`, `Dropzone`/`TextArea`/`CopyButton`/`Alert`/`ProgressBar`. + +## Global Constraints + +- **Branch:** `feat/image-ocr` (already created off `main`). Do NOT work on `main`/`develop`. +- **Scope:** Phase 1, **English only**. No structured receipt parsing, no multi-language picker, no batch. (Spec §"Out of Scope"). +- **Privacy:** the image never leaves the browser; only model files are fetched from a CDN and cached. UI must disclose the one-time model download. (Spec §Constraints). +- **Preprocessing default:** "Clean up image" is **OFF by default**. (Spec §Locked Decision 5). +- **Errors are reason-specific:** every surfaced failure names its cause via `OcrError.reason` + message. (Spec §Error Handling). +- **Test style:** Vitest. Pure transforms tested with plain `ImageData` objects in jsdom (no real canvas), mirroring `src/tools/image/mono.lib.test.ts`. Heavy deps (SDK, pdf renderer) are mocked with `vi.mock`, mirroring `src/services/capture/tauri.test.ts`. +- **Lint:** must pass `npm run lint` (no `any` in new source; use real types). +- **Tool naming:** display name **"Image to Text (OCR)"**, id `image-ocr`, category `Image`, status `beta`. + +## File Structure + +| File | Responsibility | +|------|----------------| +| `src/tools/image/ocr-preprocess.lib.ts` (create) | Pure `ImageData` transforms: `applyCleanup`, `rotate90`, `crop`. | +| `src/tools/image/ocr-preprocess.lib.test.ts` (create) | Deterministic pixel tests for the above. | +| `src/tools/image/ocr-pdf.lib.ts` (create) | `getPdfPageCount(file)`, `renderPdfPage(file, page, scale)` — wraps `openPdfRenderer`. | +| `src/tools/image/ocr-pdf.lib.test.ts` (create) | Tests mocking `@/tools/pdf/render.lib`. | +| `src/tools/image/ocr.engine.ts` (create) | The ONLY file importing the OCR SDK. Exports `createEngine()` → `{ backend, recognize }`. Smoke-tested only. | +| `src/tools/image/ocr.lib.ts` (create) | `recognize(canvas)`, engine caching, `OcrError`, reading-order sort/join, error mapping. | +| `src/tools/image/ocr.lib.test.ts` (create) | Tests mocking `./ocr.engine`. | +| `src/islands/image/ImageOcr.tsx` (create) | Thin UI orchestrator. | +| `src/registry/tools.ts` (modify) | Register the tool. | +| `astro.config.mjs` (modify) | Keep OCR SDK chunk out of the PWA precache. | + +--- + +### Task 1: Preprocessing transforms (`ocr-preprocess.lib.ts`) + +Pure, canvas-free `ImageData` transforms reused by the review-&-adjust step. No heavy deps — pure functions, fully unit-tested. + +**Files:** +- Create: `src/tools/image/ocr-preprocess.lib.ts` +- Test: `src/tools/image/ocr-preprocess.lib.test.ts` + +**Interfaces:** +- Consumes: `toGrayscale`, `toBlackWhite` from `src/tools/image/mono.lib.ts` — signatures: `toGrayscale(src: ImageData): ImageData`, `toBlackWhite(src: ImageData, threshold: number): ImageData`. +- Produces: + - `interface OcrBox { x: number; y: number; width: number; height: number }` + - `applyCleanup(src: ImageData, opts?: { threshold?: number }): ImageData` — grayscale; if `threshold` given, then hard black/white. + - `rotate90(src: ImageData): ImageData` — one 90° clockwise turn. + - `crop(src: ImageData, region: OcrBox): ImageData` — copy a clamped sub-rectangle. + +- [ ] **Step 1: Write the failing test** + +Create `src/tools/image/ocr-preprocess.lib.test.ts`: + +```ts +import { describe, it, expect } from 'vitest'; +import { applyCleanup, rotate90, crop } from './ocr-preprocess.lib'; + +// Plain ImageData is enough in jsdom — transforms only read width/height/data. +function makeImageData(width: number, height: number, px: number[][]): ImageData { + const data = new Uint8ClampedArray(px.length * 4); + px.forEach(([r, g, b, a], i) => { + data[i * 4] = r; data[i * 4 + 1] = g; data[i * 4 + 2] = b; data[i * 4 + 3] = a ?? 255; + }); + return { width, height, data, colorSpace: 'srgb' } as ImageData; +} + +describe('applyCleanup', () => { + it('grayscales when no threshold (R=G=B, alpha kept)', () => { + const out = applyCleanup(makeImageData(2, 1, [[255, 0, 0, 255], [0, 0, 255, 128]])); + expect(out.data[0]).toBe(out.data[1]); + expect(out.data[1]).toBe(out.data[2]); + expect(out.data[3]).toBe(255); + expect(out.data[7]).toBe(128); // alpha preserved + }); + + it('binarizes to 0/255 when a threshold is given', () => { + const out = applyCleanup(makeImageData(2, 1, [[100, 100, 100, 255], [200, 200, 200, 255]]), { threshold: 128 }); + expect(out.data[0]).toBe(0); // lum 100 < 128 + expect(out.data[4]).toBe(255); // lum 200 >= 128 + }); +}); + +describe('rotate90', () => { + it('turns a 2x1 into a 1x2 and moves pixels clockwise', () => { + // source row: [A=red, B=green]; 90° CW -> column top=A, bottom=B + const src = makeImageData(2, 1, [[255, 0, 0, 255], [0, 255, 0, 255]]); + const out = rotate90(src); + expect(out.width).toBe(1); + expect(out.height).toBe(2); + expect([out.data[0], out.data[1], out.data[2]]).toEqual([255, 0, 0]); // top = A + expect([out.data[4], out.data[5], out.data[6]]).toEqual([0, 255, 0]); // bottom = B + }); + + it('applied four times returns the original', () => { + const src = makeImageData(2, 1, [[1, 2, 3, 255], [4, 5, 6, 255]]); + let out = src; + for (let i = 0; i < 4; i++) out = rotate90(out); + expect(out.width).toBe(2); + expect(out.height).toBe(1); + expect(Array.from(out.data)).toEqual(Array.from(src.data)); + }); +}); + +describe('crop', () => { + it('copies the requested sub-rectangle', () => { + // 2x2: TL=1,TR=2,BL=3,BR=4 (red channel encodes id) + const src = makeImageData(2, 2, [[1, 0, 0, 255], [2, 0, 0, 255], [3, 0, 0, 255], [4, 0, 0, 255]]); + const out = crop(src, { x: 1, y: 0, width: 1, height: 2 }); // right column + expect(out.width).toBe(1); + expect(out.height).toBe(2); + expect(out.data[0]).toBe(2); // TR + expect(out.data[4]).toBe(4); // BR + }); + + it('clamps a region that exceeds bounds', () => { + const src = makeImageData(2, 1, [[1, 0, 0, 255], [2, 0, 0, 255]]); + const out = crop(src, { x: 0, y: 0, width: 99, height: 99 }); + expect(out.width).toBe(2); + expect(out.height).toBe(1); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest run src/tools/image/ocr-preprocess.lib.test.ts` +Expected: FAIL — `applyCleanup`/`rotate90`/`crop` are not defined (module not found). + +- [ ] **Step 3: Write the implementation** + +Create `src/tools/image/ocr-preprocess.lib.ts`: + +```ts +import { toGrayscale, toBlackWhite } from './mono.lib'; + +export interface OcrBox { + x: number; + y: number; + width: number; + height: number; +} + +function emptyLike(width: number, height: number): ImageData { + return { width, height, data: new Uint8ClampedArray(width * height * 4), colorSpace: 'srgb' } as ImageData; +} + +/** Grayscale; if a threshold is supplied, hard-binarize to black/white. */ +export function applyCleanup(src: ImageData, opts: { threshold?: number } = {}): ImageData { + const gray = toGrayscale(src); + return opts.threshold === undefined ? gray : toBlackWhite(gray, opts.threshold); +} + +/** One 90° clockwise rotation. dest[x'=h-1-y, y'=x]. */ +export function rotate90(src: ImageData): ImageData { + const { width: w, height: h } = src; + const out = emptyLike(h, w); // dimensions swap + for (let y = 0; y < h; y++) { + for (let x = 0; x < w; x++) { + const si = (y * w + x) * 4; + const dx = h - 1 - y; + const dy = x; + const di = (dy * h + dx) * 4; + out.data[di] = src.data[si]; + out.data[di + 1] = src.data[si + 1]; + out.data[di + 2] = src.data[si + 2]; + out.data[di + 3] = src.data[si + 3]; + } + } + return out; +} + +/** Copy a sub-rectangle, clamped to the source bounds. */ +export function crop(src: ImageData, region: OcrBox): ImageData { + const x0 = Math.max(0, Math.min(region.x, src.width)); + const y0 = Math.max(0, Math.min(region.y, src.height)); + const x1 = Math.max(x0, Math.min(region.x + region.width, src.width)); + const y1 = Math.max(y0, Math.min(region.y + region.height, src.height)); + const w = x1 - x0; + const h = y1 - y0; + const out = emptyLike(w, h); + for (let y = 0; y < h; y++) { + for (let x = 0; x < w; x++) { + const si = ((y0 + y) * src.width + (x0 + x)) * 4; + const di = (y * w + x) * 4; + out.data[di] = src.data[si]; + out.data[di + 1] = src.data[si + 1]; + out.data[di + 2] = src.data[si + 2]; + out.data[di + 3] = src.data[si + 3]; + } + } + return out; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest run src/tools/image/ocr-preprocess.lib.test.ts` +Expected: PASS (3 describes, 6 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/tools/image/ocr-preprocess.lib.ts src/tools/image/ocr-preprocess.lib.test.ts +git commit -m "feat(ocr): pure image preprocessing transforms (cleanup/rotate/crop)" +``` + +--- + +### Task 2: PDF page rasterization adapter (`ocr-pdf.lib.ts`) + +Wraps the existing `openPdfRenderer` so the island can turn a PDF page into an image blob, reusing the same `pdfjs-dist` path the `pdf-to-image` tool uses. + +**Files:** +- Create: `src/tools/image/ocr-pdf.lib.ts` +- Test: `src/tools/image/ocr-pdf.lib.test.ts` + +**Interfaces:** +- Consumes: `openPdfRenderer` from `src/tools/pdf/render.lib.ts` — `openPdfRenderer(data: ArrayBuffer | Uint8Array): Promise` where `PdfRenderer = { pageCount: number; renderPage(pageNumber: number, scale: number, mimeType?: string, quality?: number): Promise<{ blob: Blob; width: number; height: number }>; destroy(): void }`. +- Produces: + - `getPdfPageCount(file: File): Promise` + - `renderPdfPage(file: File, page: number, scale?: number): Promise` — 1-indexed `page`, default `scale = 2`. + +- [ ] **Step 1: Write the failing test** + +Create `src/tools/image/ocr-pdf.lib.test.ts`: + +```ts +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const renderPage = vi.fn(); +const destroy = vi.fn(); +const openPdfRenderer = vi.fn(); + +vi.mock('@/tools/pdf/render.lib', () => ({ + openPdfRenderer: (...args: unknown[]) => openPdfRenderer(...args), +})); + +import { getPdfPageCount, renderPdfPage } from './ocr-pdf.lib'; + +function fakeFile(): File { + // arrayBuffer() is what the lib calls; jsdom File supports it. + return new File([new Uint8Array([1, 2, 3])], 'r.pdf', { type: 'application/pdf' }); +} + +beforeEach(() => { + renderPage.mockReset(); + destroy.mockReset(); + openPdfRenderer.mockReset(); + openPdfRenderer.mockResolvedValue({ pageCount: 3, renderPage, destroy }); +}); + +describe('getPdfPageCount', () => { + it('returns the renderer page count and tears down', async () => { + expect(await getPdfPageCount(fakeFile())).toBe(3); + expect(destroy).toHaveBeenCalledOnce(); + }); +}); + +describe('renderPdfPage', () => { + it('renders the given 1-indexed page at default scale 2 and returns the blob', async () => { + const blob = new Blob(['x'], { type: 'image/png' }); + renderPage.mockResolvedValue({ blob, width: 10, height: 20 }); + const out = await renderPdfPage(fakeFile(), 2); + expect(out).toBe(blob); + expect(renderPage).toHaveBeenCalledWith(2, 2, 'image/png'); + expect(destroy).toHaveBeenCalledOnce(); + }); + + it('tears down even if rendering throws', async () => { + renderPage.mockRejectedValue(new Error('boom')); + await expect(renderPdfPage(fakeFile(), 1)).rejects.toThrow('boom'); + expect(destroy).toHaveBeenCalledOnce(); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest run src/tools/image/ocr-pdf.lib.test.ts` +Expected: FAIL — module `./ocr-pdf.lib` / its exports not found. + +- [ ] **Step 3: Write the implementation** + +Create `src/tools/image/ocr-pdf.lib.ts`: + +```ts +import { openPdfRenderer } from '@/tools/pdf/render.lib'; + +/** Number of pages in a PDF file. */ +export async function getPdfPageCount(file: File): Promise { + const renderer = await openPdfRenderer(await file.arrayBuffer()); + try { + return renderer.pageCount; + } finally { + renderer.destroy(); + } +} + +/** Rasterize one 1-indexed PDF page to a PNG blob. */ +export async function renderPdfPage(file: File, page: number, scale = 2): Promise { + const renderer = await openPdfRenderer(await file.arrayBuffer()); + try { + const { blob } = await renderer.renderPage(page, scale, 'image/png'); + return blob; + } finally { + renderer.destroy(); + } +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest run src/tools/image/ocr-pdf.lib.test.ts` +Expected: PASS (2 describes, 3 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/tools/image/ocr-pdf.lib.ts src/tools/image/ocr-pdf.lib.test.ts +git commit -m "feat(ocr): PDF page rasterization adapter over openPdfRenderer" +``` + +--- + +### Task 3: OCR engine wrapper + result/error logic (`ocr.engine.ts` + `ocr.lib.ts`) + +`ocr.engine.ts` is the sole SDK boundary (smoke-tested). `ocr.lib.ts` holds all testable logic: engine caching, reading-order assembly, and reason-specific `OcrError`s. + +**Files:** +- Create: `src/tools/image/ocr.engine.ts` +- Create: `src/tools/image/ocr.lib.ts` +- Test: `src/tools/image/ocr.lib.test.ts` +- Modify: `package.json` (add `ppu-paddle-ocr` dependency) + +**Interfaces:** +- Consumes: `OcrBox` from `./ocr-preprocess.lib`. +- Produces (from `ocr.engine.ts`): + - `type OcrBackend = 'webgpu' | 'wasm'` + - `interface RawLine { text: string; box: OcrBox; confidence: number }` + - `interface OcrEngine { backend: OcrBackend; recognize(canvas: HTMLCanvasElement): Promise }` + - `createEngine(): Promise` +- Produces (from `ocr.lib.ts`): + - `type OcrReason = 'engine-unsupported' | 'model-download' | 'inference' | 'no-text' | 'input'` + - `class OcrError extends Error { reason: OcrReason }` + - `interface OcrLine extends RawLine {}` and `interface OcrResult { text: string; lines: OcrLine[]; backend: OcrBackend }` + - `getEngine(): Promise` (cached; failure clears the cache so Retry can re-init) + - `recognize(canvas: HTMLCanvasElement): Promise` + +- [ ] **Step 1: Vertical-slice eval — pin the SDK and confirm its output shape** + +Install the engine and confirm it loads in the browser build before writing the adapter: + +```bash +npm i ppu-paddle-ocr --legacy-peer-deps +``` + +Then, in `ocr.engine.ts` (next step), the adapter maps the SDK's per-line output to `RawLine`. Verify the real shape once with a scratch check in the browser (or the package's README/types): the SDK returns detected regions with a `text` string, a quadrilateral/box, and a `score`. Map: `text` → `text`; bounding box of the region → `{x,y,width,height}`; `score` (0..1) → `confidence`. If the field names differ, adjust ONLY the mapping inside `createEngine` — nothing else in the plan depends on the SDK's native shape. + +> Note: `ppu-paddle-ocr` pulls `onnxruntime-web`, which is already a dependency. If install surfaces a peer conflict, the repo's committed `.npmrc` (`legacy-peer-deps=true`) already covers it. + +- [ ] **Step 2: Write `ocr.engine.ts` (SDK boundary, no unit test)** + +Create `src/tools/image/ocr.engine.ts`: + +```ts +import type { OcrBox } from './ocr-preprocess.lib'; + +export type OcrBackend = 'webgpu' | 'wasm'; + +export interface RawLine { + text: string; + box: OcrBox; + confidence: number; +} + +export interface OcrEngine { + backend: OcrBackend; + recognize(canvas: HTMLCanvasElement): Promise; +} + +// Convert a PP-OCR quadrilateral ([[x,y]... ] ) to an axis-aligned box. +function quadToBox(points: number[][]): OcrBox { + const xs = points.map((p) => p[0]); + const ys = points.map((p) => p[1]); + const x = Math.min(...xs); + const y = Math.min(...ys); + return { x, y, width: Math.max(...xs) - x, height: Math.max(...ys) - y }; +} + +/** + * Create the on-device OCR engine (English). This is the only file that touches + * the OCR SDK; keep it thin so the SDK stays swappable. WebGPU is used when the + * browser exposes it, otherwise the SDK falls back to WASM. + */ +export async function createEngine(): Promise { + const backend: OcrBackend = typeof navigator !== 'undefined' && 'gpu' in navigator ? 'webgpu' : 'wasm'; + const { default: PaddleOcr } = await import('ppu-paddle-ocr/web'); + const ocr = await PaddleOcr.initialize(); + + return { + backend, + async recognize(canvas: HTMLCanvasElement): Promise { + const result = await ocr.recognize(canvas); + // result.lines: Array<{ text: string; points: number[][]; score: number }> + return (result.lines ?? []).map((l: { text: string; points: number[][]; score: number }) => ({ + text: l.text, + box: quadToBox(l.points), + confidence: l.score, + })); + }, + }; +} +``` + +> If Step 1's eval showed different SDK field names (`box` instead of `points`, `confidence` instead of `score`, a top-level array instead of `.lines`), adjust the mapping above accordingly. This is the single reconciliation point. + +- [ ] **Step 3: Write the failing test for `ocr.lib.ts`** + +Create `src/tools/image/ocr.lib.test.ts`: + +```ts +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const createEngine = vi.fn(); +vi.mock('./ocr.engine', () => ({ createEngine: () => createEngine() })); + +import { recognize, getEngine, OcrError } from './ocr.lib'; + +function line(text: string, x: number, y: number, confidence = 0.9) { + return { text, box: { x, y, width: 40, height: 10 }, confidence }; +} + +beforeEach(() => { + createEngine.mockReset(); +}); + +describe('recognize', () => { + it('joins lines in reading order (top-to-bottom, then left-to-right)', async () => { + const engineRecognize = vi.fn().mockResolvedValue([ + line('world', 60, 0), + line('hello', 0, 0), + line('again', 0, 40), + ]); + createEngine.mockResolvedValue({ backend: 'webgpu', recognize: engineRecognize }); + const res = await recognize({} as HTMLCanvasElement); + expect(res.text).toBe('hello world\nagain'); + expect(res.backend).toBe('webgpu'); + expect(res.lines).toHaveLength(3); + }); + + it('throws OcrError(no-text) when nothing is detected', async () => { + createEngine.mockResolvedValue({ backend: 'wasm', recognize: vi.fn().mockResolvedValue([]) }); + await expect(recognize({} as HTMLCanvasElement)).rejects.toMatchObject({ reason: 'no-text' }); + }); + + it('wraps an inference failure as OcrError(inference) with the cause', async () => { + createEngine.mockResolvedValue({ backend: 'wasm', recognize: vi.fn().mockRejectedValue(new Error('kernel died')) }); + await expect(recognize({} as HTMLCanvasElement)).rejects.toMatchObject({ + reason: 'inference', + message: expect.stringContaining('kernel died'), + }); + }); +}); + +describe('getEngine init errors', () => { + it('maps a network/fetch failure to model-download', async () => { + createEngine.mockRejectedValueOnce(new Error('Failed to fetch model')); + await expect(getEngine()).rejects.toMatchObject({ reason: 'model-download' }); + }); + + it('maps other init failures to engine-unsupported and clears the cache for retry', async () => { + createEngine.mockRejectedValueOnce(new Error('no wasm SIMD')); + await expect(getEngine()).rejects.toMatchObject({ reason: 'engine-unsupported' }); + // cache cleared: a second call re-invokes createEngine (now succeeding) + createEngine.mockResolvedValueOnce({ backend: 'wasm', recognize: vi.fn() }); + await expect(getEngine()).resolves.toMatchObject({ backend: 'wasm' }); + expect(createEngine).toHaveBeenCalledTimes(2); + }); +}); + +it('OcrError carries name and reason', () => { + const e = new OcrError('input', 'bad file'); + expect(e).toBeInstanceOf(Error); + expect(e.name).toBe('OcrError'); + expect(e.reason).toBe('input'); +}); +``` + +- [ ] **Step 4: Run the test to verify it fails** + +Run: `npx vitest run src/tools/image/ocr.lib.test.ts` +Expected: FAIL — `./ocr.lib` exports not found. + +- [ ] **Step 5: Write the implementation** + +Create `src/tools/image/ocr.lib.ts`: + +```ts +import { createEngine, type OcrEngine, type RawLine, type OcrBackend } from './ocr.engine'; + +export type OcrReason = 'engine-unsupported' | 'model-download' | 'inference' | 'no-text' | 'input'; + +export class OcrError extends Error { + reason: OcrReason; + constructor(reason: OcrReason, message: string) { + super(message); + this.name = 'OcrError'; + this.reason = reason; + } +} + +export type OcrLine = RawLine; +export interface OcrResult { + text: string; + lines: OcrLine[]; + backend: OcrBackend; +} + +const NO_TEXT_MSG = 'No text detected — try turning on Clean up image, or use a clearer/tighter crop.'; + +function messageOf(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +// Init failures: a network/fetch problem means the model download failed; +// anything else means the engine can't run in this browser. +function toInitError(err: unknown): OcrError { + const msg = messageOf(err).toLowerCase(); + if (msg.includes('fetch') || msg.includes('network') || msg.includes('download')) { + return new OcrError( + 'model-download', + 'Couldn’t download the OCR model (network error or blocked). First use needs a connection — check it and retry.', + ); + } + return new OcrError( + 'engine-unsupported', + 'On-device OCR couldn’t start: this browser can’t run the OCR engine (missing WebAssembly SIMD support).', + ); +} + +let enginePromise: Promise | null = null; + +/** Lazily create the engine once. On failure, clear the cache so a retry re-inits. */ +export function getEngine(): Promise { + if (!enginePromise) { + enginePromise = createEngine().catch((err) => { + enginePromise = null; + throw toInitError(err); + }); + } + return enginePromise; +} + +// Reading order: group into rows by vertical overlap, then left-to-right within a row. +function sortReadingOrder(lines: RawLine[]): RawLine[] { + return [...lines].sort((a, b) => { + const rowTol = Math.min(a.box.height, b.box.height) * 0.5; + if (Math.abs(a.box.y - b.box.y) > rowTol) return a.box.y - b.box.y; + return a.box.x - b.box.x; + }); +} + +/** Recognize text in a prepared canvas. Throws OcrError with a specific reason. */ +export async function recognize(canvas: HTMLCanvasElement): Promise { + const engine = await getEngine(); + let raw: RawLine[]; + try { + raw = await engine.recognize(canvas); + } catch (err) { + throw new OcrError('inference', `OCR failed: ${messageOf(err)}`); + } + const lines = sortReadingOrder(raw); + if (lines.length === 0) throw new OcrError('no-text', NO_TEXT_MSG); + return { text: lines.map((l) => l.text).join('\n'), lines, backend: engine.backend }; +} +``` + +- [ ] **Step 6: Run the tests to verify they pass** + +Run: `npx vitest run src/tools/image/ocr.lib.test.ts` +Expected: PASS (3 describes + standalone test, 6 tests). + +> Note: `ocr.engine.ts` has no unit test (it is the SDK boundary); it is exercised by the manual smoke checklist in Task 4. + +- [ ] **Step 7: Commit** + +```bash +git add package.json package-lock.json src/tools/image/ocr.engine.ts src/tools/image/ocr.lib.ts src/tools/image/ocr.lib.test.ts +git commit -m "feat(ocr): engine wrapper, reading-order assembly, reason-specific errors" +``` + +--- + +### Task 4: Orchestrator island + registry + PWA config (`ImageOcr.tsx`) + +Wires input (image or PDF) → review-&-adjust (rotate + optional cleanup/threshold) → run → results (editable text, copy, download). Registers the tool and keeps the OCR chunk out of the PWA precache. Deliverable: a working, reachable tool. + +**Files:** +- Create: `src/islands/image/ImageOcr.tsx` +- Modify: `src/registry/tools.ts` +- Modify: `astro.config.mjs` + +**Interfaces:** +- Consumes: `recognize`, `OcrError`, `type OcrResult` from `@/tools/image/ocr.lib`; `applyCleanup`, `rotate90` from `@/tools/image/ocr-preprocess.lib`; `getPdfPageCount`, `renderPdfPage` from `@/tools/image/ocr-pdf.lib`; `downloadService.download(blob, filename)`; `usePasteImage`; UI: `Dropzone`, `Button`, `Alert`, `TextArea`, `CopyButton`. (No `ProgressBar` — it is determinate-only; OCR uses an indeterminate busy indicator.) +- Produces: default-exported React component `ImageOcr`. + +- [ ] **Step 1: Register the tool (make the route resolvable)** + +In `src/registry/tools.ts`, add this entry to the tools array near the other `image-*` entries (e.g. after `image-face-blur`). Import the icon `ScanText` from `lucide-react` in the existing icon import block: + +```ts + { + id: 'image-ocr', + name: 'Image to Text (OCR)', + category: 'Image', + route: '/tools/image-ocr', + keywords: ['ocr', 'receipt', 'scan', 'text', 'extract', 'recognize', 'read', 'document'], + icon: ScanText, + summary: 'Extract text from an image or PDF with on-device AI', + load: () => import('@/islands/image/ImageOcr'), + status: 'beta' + }, +``` + +- [ ] **Step 2: Keep the OCR SDK chunk out of the PWA precache** + +In `astro.config.mjs`, add the OCR SDK chunk glob to the existing `workbox.globIgnores` array (same list that already ignores the Monaco/DbDiagram chunks): + +```js + '**/ppu-paddle-ocr*.js', + '**/ort-*.wasm', +``` + +- [ ] **Step 3: Implement the island** + +Create `src/islands/image/ImageOcr.tsx`: + +```tsx +import { useEffect, useRef, useState } from 'react'; +import { Dropzone } from '@/components/ui/Dropzone'; +import { Button } from '@/components/ui/Button'; +import { Alert } from '@/components/ui/Alert'; +import { TextArea } from '@/components/ui/TextArea'; +import { CopyButton } from '@/components/ui/CopyButton'; +import { usePasteImage } from '@/hooks/usePasteImage'; +import { downloadService } from '@/services/download'; +import { applyCleanup, rotate90 } from '@/tools/image/ocr-preprocess.lib'; +import { recognize, OcrError } from '@/tools/image/ocr.lib'; +import { getPdfPageCount, renderPdfPage } from '@/tools/image/ocr-pdf.lib'; + +// Oversized inputs are downscaled before inference (memory/perf guard; Spec §Error Handling). +const MAX_DIM = 2000; + +// Draw a blob to an offscreen canvas (downscaled if huge) and return its ImageData (the "base"). +async function blobToImageData(blob: Blob): Promise { + const bitmap = await createImageBitmap(blob); + const scale = Math.min(1, MAX_DIM / Math.max(bitmap.width, bitmap.height)); + const w = Math.max(1, Math.round(bitmap.width * scale)); + const h = Math.max(1, Math.round(bitmap.height * scale)); + const canvas = document.createElement('canvas'); + canvas.width = w; + canvas.height = h; + const ctx = canvas.getContext('2d'); + if (!ctx) throw new Error('Canvas is not supported in this browser'); + ctx.drawImage(bitmap, 0, 0, w, h); + bitmap.close?.(); + return ctx.getImageData(0, 0, w, h); +} + +// Apply rotation + optional cleanup, returning a canvas ready for OCR/preview. +function buildAdjusted( + base: ImageData, + opts: { quarters: number; cleanup: boolean; threshold: number }, +): HTMLCanvasElement { + let img = base; + for (let i = 0; i < opts.quarters; i++) img = rotate90(img); + if (opts.cleanup) img = applyCleanup(img, { threshold: opts.threshold }); + const canvas = document.createElement('canvas'); + canvas.width = img.width; + canvas.height = img.height; + const ctx = canvas.getContext('2d'); + if (!ctx) throw new Error('Canvas is not supported in this browser'); + const sink = ctx.createImageData(img.width, img.height); + sink.data.set(img.data); + ctx.putImageData(sink, 0, 0); + return canvas; +} + +export default function ImageOcr() { + const [file, setFile] = useState(null); + const [isPdf, setIsPdf] = useState(false); + const [pageCount, setPageCount] = useState(0); + const [page, setPage] = useState(1); + const [base, setBase] = useState(null); + + const [quarters, setQuarters] = useState(0); + const [cleanup, setCleanup] = useState(false); // OFF by default (Global Constraints) + const [threshold, setThreshold] = useState(140); + + const [busy, setBusy] = useState(false); + const [text, setText] = useState(''); + const [backendNote, setBackendNote] = useState(''); + const [error, setError] = useState(''); + const [retryable, setRetryable] = useState(false); + + const previewRef = useRef(null); + + const reset = () => { + setBase(null); setText(''); setError(''); setRetryable(false); + setQuarters(0); setCleanup(false); setPage(1); setPageCount(0); + }; + + const onDrop = async (files: File[]) => { + const f = files.find((x) => x.type.startsWith('image/') || x.type === 'application/pdf') ?? null; + reset(); + setFile(f); + if (!f) return; + const pdf = f.type === 'application/pdf'; + setIsPdf(pdf); + try { + if (pdf) { + const count = await getPdfPageCount(f); + setPageCount(count); + setBase(await blobToImageData(await renderPdfPage(f, 1))); + } else { + setBase(await blobToImageData(f)); + } + } catch (e) { + setError(e instanceof Error ? e.message : 'Could not load that file.'); + } + }; + usePasteImage((f) => onDrop([f])); + + // Load a different PDF page. + useEffect(() => { + if (!file || !isPdf || pageCount === 0) return; + let alive = true; + renderPdfPage(file, page) + .then(blobToImageData) + .then((d) => alive && setBase(d)) + .catch((e) => alive && setError(e instanceof Error ? e.message : 'Could not render that page.')); + return () => { alive = false; }; + }, [file, isPdf, page, pageCount]); + + // Live preview of the adjusted image. + useEffect(() => { + if (!base || !previewRef.current) return; + const canvas = buildAdjusted(base, { quarters, cleanup, threshold }); + const el = previewRef.current; + el.width = canvas.width; + el.height = canvas.height; + el.getContext('2d')?.drawImage(canvas, 0, 0); + }, [base, quarters, cleanup, threshold]); + + const runOcr = async () => { + if (!base) return; + setBusy(true); setError(''); setRetryable(false); setText(''); + try { + const canvas = buildAdjusted(base, { quarters, cleanup, threshold }); + const result = await recognize(canvas); + setText(result.text); + setBackendNote( + result.backend === 'wasm' + ? 'Ran in slower CPU (WASM) mode — WebGPU isn’t available in this browser.' + : '', + ); + } catch (e) { + if (e instanceof OcrError) { + setError(e.message); + setRetryable(e.reason === 'model-download'); + } else { + setError(e instanceof Error ? e.message : 'OCR failed.'); + } + } finally { + setBusy(false); + } + }; + + const outName = (file?.name.replace(/\.[^.]+$/, '') || 'ocr') + '.txt'; + const download = () => downloadService.download(new Blob([text], { type: 'text/plain' }), outName); + + return ( +
+ +
+

Drop an image or PDF, or click to browse

+

Runs on-device · or paste (⌘V). First use downloads the OCR model once.

+
+ + + {file &&

{file.name}

} + + {isPdf && pageCount > 1 && ( +
+ + Page {page} / {pageCount} + +
+ )} + + {base && ( +
+ + +
+ + + {cleanup && ( + + )} + +
+ + {busy && ( +

+ Extracting text… (first run downloads the OCR model once) +

+ )} +
+ )} + + {error && ( + + {error} + {retryable && <> } + + )} + + {text && ( +
+ {backendNote &&

{backendNote}

} +