A hybrid OCR pipeline built for the MOSIP Hackathon, designed to extract and verify identity data from scanned documents and images. The system intelligently switches between printed OCR and AI-based handwritten OCR, with a modern frontend for review and verification.
- Hybrid OCR approach (Printed + Handwritten)
- Automatic document type detection
- Supports images and PDFs
- Designed for identity documents (ID cards, Aadhaar, PAN, passports, MOSIP Blue Card samples)
- Built with Node.js backend and React + Vite frontend
Frontend (React + Vite)
↓
Backend (Express API)
↓
Hybrid OCR Engine
├── Tesseract.js (Printed Text)
└── TrOCR (Handwritten Text - ONNX)
- Node.js (v18+ recommended)
- npm
cd Backend
npm install
npm run serverBackend runs the OCR engine, extraction logic, and verification routes.
cd Frontend
npm install
npm run devFrontend runs on Vite and connects to the backend APIs.
The OCR engine automatically decides the best approach:
- Uses Tesseract.js
- Keyword detection + confidence analysis
- Optimized for government-issued IDs
- Uses TrOCR (Xenova transformers)
- ONNX models cached locally
- Activated when printed confidence is low or handwriting is detected
- JPG / PNG / WEBP
- PDF (embedded text or scanned PDFs)
- Upload document
- View extracted fields
- Raw OCR output viewer
- OCR confidence & model used
- Field-level verification
- Confidence indicators
- Designed for manual review workflows
| Extraction Panel | Verification Panel |
|---|---|
 |
 |
| Extraction Test | Verification Test |
|---|---|
 |
 |
- Automatic PDF to image conversion
- Image preprocessing (grayscale, safe transforms)
- Printed vs handwritten decision logic
- Strip-based OCR support for complex layouts
- Modular routes:
- /extraction
- /verification
- /layouts
- React
- TypeScript
- Vite
- Tailwind / ShadCN UI
- Node.js
- Express
- Tesseract.js
- TrOCR (Xenova Transformers)
- ONNX Runtime
MIT License