🧠 Project 1A: PDF Outline Extractor by Shakti Pixels

A powerful, fully offline, multilingual Python-based system for generating structured outlines and titles from PDF documents using intelligent text extraction, NLP-driven heading analysis, and hierarchical document structuring.

---

🎯 Overview

This project provides an advanced solution for processing PDFs to automatically generate well-structured outlines and extract meaningful document titles. It supports 15+ languages and works entirely offline.

✨ Core Capabilities:

• Smart Text Extraction using PyMuPDF (fitz) for robust layout analysis
• Multilingual Language Detection via SpaCy and custom heuristics
• Heuristic Heading Classification using font properties, position, and NLP
• Hierarchical Outline Construction from heading relationships
• Automated Title Generation through metadata and content introspection

---

🚀 Quick Start (Local)

1. Add Your PDFs

Place .pdf files inside the inputs/ folder.

2. Run the Main Processor

python main.py

3. Check Output Results

The system generates .json output files in the outputs/ folder with structure:

{
  "title": "Extracted Title",
  "outline": [
    {"level": 1, "text": "Heading 1"},
    {"level": 2, "text": "Subheading 1.1"},
    ...
  ]
}

---

🐳 Docker-Based Deployment (Recommended)

🔨 Build the Docker Image

docker build -t pdf-outline-extractor .

📂 Ensure Required Directories

mkdir -p inputs outputs

▶️ Run the Docker Container

• Linux/macOS:

docker run -v $(pwd)/inputs:/app/inputs -v $(pwd)/outputs:/app/outputs pdf-outline-extractor

• Windows PowerShell:

docker run -v ${PWD}/inputs:/app/inputs -v ${PWD}/outputs:/app/outputs pdf-outline-extractor

• Windows CMD:

docker run -v %cd%/inputs:/app/inputs -v %cd%/outputs:/app/outputs pdf-outline-extractor

---

🧠 Processing Pipeline

1. Initial Sampling — Analyzes first few pages for language detection
2. Language Detection — SpaCy-powered model selection
3. Full Text Extraction — Font size, layout, and block merging
4. Heading Classification — Dynamic scoring with multi-feature heuristics
5. Title Derivation — Extracted from content or metadata
6. Hierarchy Structuring — Builds outline H1→H2→H3→H4 with validation
7. Output Generation — Clean JSON output

---

🔍 Output Sample

For example.pdf, the system generates:

{
  "title": "市町村合併を考慮した市区町村パネルデータ",
  "outline": [
    {"level": "H1", "text": "市町村合併を考慮した市区町村パ...", "page": 1},
    {"level": "H2", "text": "近藤恵介", "page": 1},
    {"level": "H3", "text": "市区町村コンバータの作成⽅法", "page": 5}
  ]
}

Output Highlights

• H1–H4 level classification
• Page number tagging
• Truncated and cleaned headings
• Language-aware formatting (CJK, RTL, etc.)

---

⚙️ System Architecture

📁 Project Structure

Challenge_1a/
├── main.py
├── requirements.txt
├── Dockerfile
├── download_models.py
├── inputs/
├── outputs/
├── models/
│   ├── xx_ent_wiki_sm-3.7.0.whl
│   └── en_core_web_sm-3.7.1.whl
└── pdf_utils/
    ├── __init__.py
    ├── extract_blocks.py
    ├── language.py
    ├── classify_headings.py
    └── structure_outline.py

---

🧠 NLP & ML Techniques

Core Techniques:

• Font Feature Engineering: Font size, boldness, positioning
• Fragment Merging: Combines broken or wrapped lines
• Multi-Language Support: Devanagari, CJK, Arabic, Latin, Cyrillic
• Heading Confidence Scoring: 15+ contextual features
• Outline Logic Check: Ensures valid H1 > H2 > H3 flow

Language Detection Matrix

Script	Examples	Detection	NLP Support
Latin	English, French	✅	✅
CJK	Chinese, Japanese	✅	✅
Arabic	Arabic	✅	✅
Cyrillic	Russian	✅	✅
Devanagari	Hindi	✅	✅

---

📦 Key Dependencies

Component	Library	Version	Purpose
PDF Parsing	PyMuPDF	1.24.1	Extract text & layout
Language Detection	SpaCy + langdetect	3.7.x	Multilingual support
NLP Processing	xx_ent_wiki_sm / en_core_web_sm	3.7.x	Text analysis
ML/NLP	scikit-learn, pandas, numpy	Latest	Feature scoring
Progress & Utilities	tqdm, joblib	Latest	UX & optimization

---

🧰 Core Modules Explained

🔤 extract_blocks.py

• Uses PyMuPDF for extracting text blocks with font data
• Merges fragmented lines and removes headers/footers

🌐 language.py

• Detects primary language using SpaCy + heuristics
• Loads appropriate NLP models with caching

🔍 classify_headings.py

• Assigns heading levels using scoring model
• Analyzes font size, position, NLP content patterns

📋 structure_outline.py

• Builds title and hierarchical headings
• Ensures clean H1 > H2 > H3 relationship

---

🛠️ Configuration & Optimization

Common Errors & Fixes

• Missing Models: Run python download_models.py to reinstall
• MemoryError: Use Docker with -m 8g or process fewer files
• Corrupted PDFs: Ensure input files have extractable text
• No Headings Found: Check formatting or tweak scoring thresholds

Performance Tips

• First 5 pages used for sampling — keep relevant headings upfront
• Models loaded once per run to optimize memory
• Batch mode supported — place multiple files in inputs/

---

📈 Performance Benchmarks (Intel i7, 16GB RAM)

PDF Size	Pages	Avg. Time
Small	1–10	2–4 sec
Medium	11–30	4–8 sec
Large	31–50	8–15 sec

Accuracy:

• Heading Detection: 90–95% on structured docs
• Language Detection: >95%
• Title Extraction: 85–90% success rate
• Hierarchy Integrity: >90% H1-H2-H3 logic

---

🐳 Dockerfile Features

• Based on python:3.10-slim
• Includes .whl SpaCy models
• No internet required after build
• Memory-friendly and portable

# Basic Build
docker build -t pdf-extractor .

# Multi-stage (smaller image)
docker build --target production -t pdf-extractor:prod .

# Specific architecture
docker build --platform linux/amd64 -t pdf-extractor .

---

💻 System Requirements

• OS: Linux/macOS/Windows (Docker recommended)
• Python: 3.10+
• RAM: Minimum 4GB, 8GB recommended
• Storage: ~200MB including models
• CPU: x86_64 compatible

---

📝 License

This project was developed independently by the Shakti Pixels team as part of an advanced NLP and PDF analysis challenge for Adobe India Hackathon.

For technical support or collaboration, reach out to the Shakti Pixels team!