Text capture: read printed labels that have no barcode

[MadsenDev]

Adds a Text mode to the phone scanner, for the serial on a chassis label, the Wi-Fi password on a router, the product key on a card with no barcode beside it.

Two commits: the feature, then a rework after it was tried on a real phone and found not good enough.

Scope

OCR was on the explicitly out-of-scope list in docs/roadmap.md. It is the one item that moved, and it is worth a deliberate look before merging. The reasoning: it does not pull against the product's single job — it is that job applied to a label with no barcode. The roadmap now records the move and why.

The rule everything follows

A decode is exact; a transcription is a guess. So text capture:

• is a separate mode — codes stay the default and are untouched; it never takes over when a barcode fails
• never fires on its own — aim, capture, then tap the line you want
• sends byte-for-byte what is on screen — no normalisation between choosing and sending, and no edit field
• runs entirely on the device — the frame is read once and dropped, no image leaves the phone
• travels as format OCR_TEXT, so the receiver captions it as a transcription rather than passing it off as a decode

Getting a readable frame

The camera opened at 720p and took whatever focus the device was in. Fine for barcodes — ZXing reads a soft 640px frame — but a product key slightly out of focus is not a harder read, it is a different string.

New camera.ts: continuous autofocus asked for explicitly rather than trusting the device default, tap-to-focus in both modes, 1080p at open, and the sharpest mode the device offers once text capture starts. Every control degrades to a no-op where unsupported.

The band

Text capture reads one full-width strip, about a seventh of the frame tall, drawn as brackets with everything outside dimmed. A boundary, not a hint — which is why it is never on screen beside the corner guide, which is a hint.

The drawn rectangle and the cropped region come from one constant and are never allowed to differ, not even by a forgiving margin that would read slightly more than the brackets promise. A clipped line is something the user can see and fix; a band that quietly reads more than it shows is the phone lying about what it saw. Verified in-browser to match to three decimal places at 16:9 and 4:3, with text above and below the band correctly excluded.

Text mode gives up the full-bleed preview to make that visible: under cover on a portrait phone, both ends of a wide key sit outside the preview.

Reading it three times

Rescaling the same captured band perturbs the engine enough that its mistakes stop lining up. Each candidate carries how many reads produced it — 3/3, 1/3 — which is a far better guide than the engine's own confidence, reported just as brightly for a line it got wrong.

The most common failure was not a misread character: it was a space appearing beside a hyphen with every character correct. Ranking counts whitespace-only differences as agreement. Both forms stay in the list; ranking never rewrites.

Measured

Six strings × six camera-like conditions (soft focus, glare, tilt, sensor noise, small text, sans-serif). Exact full-string recovery:

	first candidate	anywhere in list
full frame, one read	0%	36%
band, one read	33%	33%
band, three scales + agreement	39%	50%

Measured and rejected rather than assumed:

• greyscale normalisation + local (Sauvola) thresholding — no gain, glare got worse; Tesseract's own binarisation is better
• multi-frame voting on consecutive camera frames — accuracy went down; a steady phone gives near-identical frames and the engine repeats its errors exactly
• character whitelist, the 11 MB full-precision language model — neither changed a single result
• raw-line page segmentation — catastrophic, 0%

Where this leaves it

Still not good enough to call solved, and the docs say so rather than dressing it up: a random product key carries no word the engine can lean on. The next step is named in the roadmap — an edit step for the one wrong character, since the diagnosis showed the failures really are that shallow.

All of the above is the WASM engine, which is what a headless browser exposes. Android's TextDetector is ML Kit and should be better on exactly this workload, so the primary target is likely better than these numbers. Worth confirming on real hardware — the focus fix in particular is untestable here and may be doing most of the work.

Cost

tesseract.js assets are staged from node_modules into public/ocr at build time and served from our own origin, never the CDN the library defaults to. ~14 MB in dist (git-ignored, regenerated per build); a device on the fallback path fetches ~4 MB core plus ~3 MB language model once. Phones with native TextDetector fetch none of it. workerBlobURL: false loads the worker from its path rather than a blob: URL, so no CSP relaxation was needed.

Verification

• 64 relay tests pass; typecheck and build clean
• OCR end to end in a real browser: capture → tap → Sent only after scan_ack → receiver shows the value byte-identical, captioned OCR TEXT
• barcode path re-tested including a text-and-back round trip (no double loop, no stuck state)
• relay logs confirmed to carry format=OCR_TEXT length=29 and nothing else
• fixes a latent race the mode switch would have exposed: a detect() in flight when the decode loop stops could resolve afterwards, sending one last barcode or leaving two loops running

🤖 Generated with Claude Code

https://claude.ai/code/session_012ooGSXdKf1ZKVKSiTycKFs

---

Generated by Claude Code