- Project Domain & Real-World Problem
- High-Level Project Overview
- System Architecture & Technical Workflow
- Tech Stack Breakdown
- Core Features
- Algorithms, Models & Techniques Used
- Installation & Setup
- How the Project Works
- Use Cases & Sample Scenarios
- Future Scope & Improvements
- Contributing & License
Computer Vision | Natural Language Processing | Multimodal AI | Accessibility Technology
In today's digital landscape, billions of images are uploaded daily across platforms, yet understanding and extracting actionable insights from visual content remains challenging. Users face several pain points:
- Content Accessibility: Visually impaired users struggle to understand image content
- Information Extraction: Manually identifying objects and text in images is time-consuming
- Document Analysis: Extracting text from documents, receipts, and signs requires specialized tools
- Visual Search: Users need to ask questions about image content without manual analysis
- Privacy Concerns: Cloud-based image analysis services compromise user privacy by uploading sensitive images to external servers
- Accessibility Users: Visually impaired individuals needing image descriptions
- Content Creators: Bloggers, marketers generating alt-text and captions
- Researchers & Analysts: Professionals extracting data from visual documents
- Developers: Teams needing local, privacy-preserving image analysis APIs
- Education: Students and educators analyzing historical images, diagrams, and documents
An intelligent, privacy-first web application that combines computer vision and natural language processing to analyze images comprehensively. Users upload an image and receive:
- Object Detection: Identification of all visible objects with confidence scores
- Text Extraction: OCR-based extraction of any readable text
- Intelligent Captioning: Natural language descriptions generated by AI
- Interactive Q&A: Conversational interface to ask questions about image content
- ✅ Provide multi-modal image analysis (vision + text + AI reasoning)
- ✅ Ensure 100% local processing with no external API dependencies
- ✅ Enable persistent chat history for reviewing past analyses
- ✅ Deliver real-time responses with optimized model inference
- ✅ Maintain production-grade security with rate limiting and validation
- Detects 80+ object classes (COCO dataset)
- Extracts text in multiple languages (Tesseract OCR)
- Generates context-aware captions using LLMs
- Supports conversational Q&A with memory retention
- Automatic image optimization and resource cleanup
- Rate limiting and file validation for security
┌─────────────────────────────────────────────────────────────────┐
│ CLIENT LAYER (Browser) │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ HTML/CSS/JS Frontend (Tailwind CSS + Bootstrap) │ │
│ │ - Image Upload Interface │ │
│ │ - Chat History Viewer │ │
│ │ - Real-time Q&A Interface │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
↕ HTTP/JSON
┌─────────────────────────────────────────────────────────────────┐
│ FLASK WEB SERVER (Backend) │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Route Handlers: /upload, /ask, /get_chat_history │ │
│ │ Middleware: Rate Limiting, Security Headers, Validation │ │
│ │ Session Management: Chat History (JSON File Storage) │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
↕
┌─────────────────────────────────────────────────────────────────┐
│ IMAGE PROCESSING PIPELINE │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ 1. Image Upload → Validation → Secure Storage │ │
│ │ 2. Optimization (Resize, Compress, Convert to RGB) │ │
│ │ 3. Parallel Processing: │ │
│ │ ├─ Object Detection (Faster R-CNN) │ │
│ │ ├─ OCR Text Extraction (Tesseract) │ │
│ │ └─ Caption Generation (Ollama + Gemma 2) │ │
│ │ 4. Results Aggregation & Storage │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
↕
┌─────────────────────────────────────────────────────────────────┐
│ AI/ML INFERENCE LAYER │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ PyTorch (CUDA/CPU): Faster R-CNN ResNet50-FPN │ │
│ │ Tesseract OCR: Multi-language Text Recognition │ │
│ │ Ollama Service: Gemma 2:9B LLM (Local Inference) │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
↕
┌─────────────────────────────────────────────────────────────────┐
│ STORAGE & LOGGING LAYER │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ uploads/ → Temporary image storage │ │
│ │ chat_history.json → Persistent chat sessions │ │
│ │ app.log → Application logging │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
-
Image Upload:
- User uploads image via browser → Flask receives multipart/form-data
- Security validation: file type, size (max 16MB), extension check
- Secure filename generation using
werkzeug.secure_filename - Image saved to
uploads/directory
-
Preprocessing:
- Image loaded via Pillow → converted to RGB
- Resized to max 1024x1024 (maintains aspect ratio)
- JPEG compression applied (quality=85) for optimization
- Temporary optimized version created
-
Parallel Analysis:
- Thread 1: Faster R-CNN inference (PyTorch + CUDA if available)
- Thread 2: Tesseract OCR text extraction
- Thread 3: LLM caption generation (Ollama API call)
-
Results Aggregation:
- Results combined into JSON structure
- Stored in-memory with unique
chat_id - Persisted to
chat_history.jsonfor session recovery - Base64-encoded image sent to frontend
-
Q&A Interaction:
- User question → Flask
/askendpoint - Retrieved cached image analysis from
chat_id - LLM generates context-aware response using image metadata
- Response stored in Q&A history and returned to UI
- User question → Flask
-
Cleanup & Optimization:
- Background thread runs hourly cleanup
- Removes chats older than 7 days
- Garbage collection to free memory
- Maintains max 100 chats in storage
| Technology | Purpose | Why Chosen |
|---|---|---|
| HTML5 / CSS3 | Semantic markup and styling | Standard web technologies for cross-browser compatibility |
| Tailwind CSS | Utility-first CSS framework | Rapid UI development with responsive design out-of-the-box |
| Bootstrap 5 | Component library (chat interface) | Pre-built chat UI components for professional aesthetics |
| Vanilla JavaScript | Client-side logic | Lightweight, no framework overhead, direct DOM manipulation |
| Font Awesome | Icon library | Consistent, scalable vector icons |
| Technology | Version | Purpose | Why Chosen |
|---|---|---|---|
| Flask | 3.1.2 | Web framework | Lightweight, Pythonic, perfect for ML integration |
| Werkzeug | - | WSGI utility library | Secure file handling and request parsing |
| Flask-Limiter | 4.0.0 | Rate limiting | Prevents abuse and DDoS attacks |
| Python 3.12 | - | Core language | Latest features, performance improvements |
| Model/Library | Purpose | Why Chosen |
|---|---|---|
| PyTorch | Deep learning framework | Industry-standard for CV, CUDA support, pre-trained models |
| Faster R-CNN (ResNet50-FPN) | Object detection | High accuracy (mAP ~37%), 80 COCO classes, production-ready |
| Tesseract OCR | Text extraction | Open-source, multi-language, industry-proven |
| Ollama | LLM inference server | Local model hosting, no API keys, privacy-preserving |
| Gemma 2:9B | LLM for captions/Q&A | Google's open model, balanced performance/speed |
| Pillow (PIL) | Image processing | Standard Python imaging library, extensive format support |
Note on YOLO: While the current implementation uses Faster R-CNN for superior accuracy, YOLO (You Only Look Once) is a viable alternative for real-time detection scenarios. Future versions could implement YOLO via ultralytics for faster inference at the cost of slight accuracy trade-offs.
| Storage Type | Technology | Purpose |
|---|---|---|
| Session Storage | In-memory Python dict | Fast access to active chat data |
| Persistent Storage | JSON file (chat_history.json) |
Session recovery across restarts |
| Image Storage | File system (uploads/) |
Temporary image hosting |
| Logging | Python logging module → app.log |
Error tracking and debugging |
| Tool | Purpose |
|---|---|
| Git | Version control |
| Virtual Environment (venv) | Dependency isolation |
| pip | Package management |
| CUDA Toolkit (optional) | GPU acceleration for PyTorch |
| Logging Framework | Production-grade error tracking |
-
🔍 Object Detection
- Detects 80 object classes (people, vehicles, animals, furniture, etc.)
- Confidence scores for each detection
- Powered by Faster R-CNN with ResNet50 backbone
-
📝 Text Extraction (OCR)
- Extracts printed and handwritten text
- Multi-language support (via Tesseract)
- Handles rotated and skewed text
-
🖼️ AI-Generated Captions
- Natural language descriptions of image content
- Context-aware using detected objects + extracted text
- Generated by Gemma 2:9B LLM
-
💬 Interactive Q&A
- Ask unlimited questions about uploaded images
- Conversational memory (remembers previous questions)
- Context-driven responses using image metadata
-
📜 Chat History
- Persistent storage of all chat sessions
- Retrieve past analyses by chat ID
- Automatic cleanup of old chats (7-day retention)
-
📤 Image Upload
- Drag-and-drop or click-to-upload
- Real-time preview before processing
- Supports PNG, JPG, JPEG, GIF (max 16MB)
-
🔒 Security
- Rate limiting (200/day, 50/hour per IP)
- File type and size validation
- Secure filename sanitization
- Security headers (XSS, CSRF, clickjacking protection)
-
⚡ Performance Optimization
- Image compression and resizing before processing
- LRU caching for frequently accessed data
- Background thread cleanup (memory management)
- Async processing with ThreadPoolExecutor
-
🗂️ Session Management
- Unique chat IDs for each session
- Automatic session expiry (7 days)
- Max 100 chats stored (FIFO eviction)
-
📊 Logging & Monitoring
- Structured logging (timestamp, level, message)
- File + console output
- Error tracking for debugging
-
🔄 Automatic Cleanup
- Hourly background cleanup thread
- Removes expired chats
- Deletes temporary optimized images
- Garbage collection for memory management
Model Architecture:
Input Image (3×H×W)
↓
ResNet50 Backbone (Feature Extraction)
↓
Feature Pyramid Network (Multi-scale Features)
↓
Region Proposal Network (RPN) - Generates ~2000 proposals
↓
RoI Align (Region of Interest Pooling)
↓
Classification Head → Object Class (80 COCO classes)
Regression Head → Bounding Box Coordinates
Why Faster R-CNN?
- Accuracy: mAP of ~37% on COCO dataset (superior to YOLO v3)
- Pre-trained Weights: Available in torchvision, no custom training needed
- Two-Stage Detection: Separate region proposal + classification = higher precision
- COCO Classes: Detects 80 common objects (person, car, dog, laptop, etc.)
Technical Details:
- Confidence threshold: 0.5 (filters low-confidence detections)
- Device: Automatically uses CUDA if available, else CPU
- Input: RGB images (normalized tensors)
Alternative: YOLO (You Only Look Once)
- YOLO is a single-stage detector optimized for speed
- Use case: Real-time video processing (30+ FPS)
- Trade-off: Slightly lower accuracy (~33% mAP) for 3x faster inference
- Implementation:
ultralyticslibrary with YOLOv8/v11
How It Works:
- Image preprocessing (binarization, noise removal)
- Text localization (finds text regions)
- Character segmentation
- Recognition using LSTM neural networks
- Post-processing (spell-check, language models)
Why Tesseract?
- Open-source and free
- Supports 100+ languages
- High accuracy on printed text (90%+)
- Handles complex layouts (multi-column, tables)
Configuration:
- Language: English (default), extendable to other languages
- Page Segmentation Mode: Auto-detect (PSM 3)
Model Details:
- Architecture: Gemma 2 (Google's open LLM)
- Parameters: 9 billion
- Context Window: 4096 tokens (Q&A), 2048 (captions)
- Inference: Local via Ollama (no cloud API)
Prompting Strategy:
# Caption Generation Prompt
f"""Generate a concise, descriptive caption for an image that contains
{objects_detected} and {extracted_text}. Provide a single sentence
that describes what might be in the image based on these detected elements."""
# Q&A Prompt
f"""Analyze this image and answer the question professionally:
- Caption: {caption}
- Detected Objects: {objects}
- Extracted Text: {text}
Question: {user_question}"""Why Gemma 2?
- Privacy: Runs entirely locally, no data leaves your machine
- Cost: Free (no API fees)
- Performance: Balanced quality/speed for 9B parameters
- Open Weights: No vendor lock-in
Why Ollama?
- Simplifies LLM deployment (no manual GGUF loading)
- Automatic model management (pull, update)
- RESTful API for easy integration
- Supports multiple models (Llama, Mistral, etc.)
Optimization Techniques:
# 1. Resize (maintain aspect ratio)
max_size = (1024, 1024)
img.thumbnail(max_size, Image.Resampling.LANCZOS)
# 2. Color Space Conversion
if img.mode != 'RGB':
img = img.convert('RGB')
# 3. JPEG Compression
img.save(path, 'JPEG', quality=85, optimize=True)Why These Steps?
- Resize: Reduces GPU memory usage, speeds up inference
- RGB Conversion: Ensures compatibility with PyTorch models
- Compression: Reduces storage without visible quality loss
Techniques:
-
File Validation:
- Whitelist allowed extensions (
.png,.jpg,.jpeg,.gif) - Max file size: 16MB
- Secure filename generation (prevents path traversal attacks)
- Whitelist allowed extensions (
-
Rate Limiting:
- 200 requests/day per IP
- 50 requests/hour per IP
- Prevents DDoS and abuse
-
Security Headers:
X-Content-Type-Options: nosniff(prevents MIME sniffing)X-Frame-Options: SAMEORIGIN(clickjacking protection)X-XSS-Protection: 1; mode=block(XSS filter)Strict-Transport-Security(enforces HTTPS)
Techniques:
- LRU Cache:
@lru_cache(maxsize=100)for frequently accessed data - Background Cleanup Thread: Runs every hour to:
- Delete chats older than 7 days
- Enforce max 100 chats limit (FIFO eviction)
- Run garbage collection (
gc.collect())
- Temporary File Cleanup: Deletes optimized images after processing
-
Python 3.8+ (Recommended: 3.12)
python --version # Check version -
Tesseract OCR
- Linux:
sudo apt-get install tesseract-ocr - macOS:
brew install tesseract - Windows: Download installer
- Linux:
-
Ollama (for LLM inference)
- Install from ollama.ai
- Linux/macOS:
curl -fsSL https://ollama.ai/install.sh | sh - Windows: Download from website
-
CUDA Toolkit (Optional, for GPU acceleration)
- Required only if you have NVIDIA GPU
- Download CUDA
-
Clone Repository
git clone https://github.com/Mithrajith/image_captioning.git cd image_captioning -
Create Virtual Environment
python -m venv env source env/bin/activate # Linux/macOS # OR env\Scripts\activate # Windows
-
Install Python Dependencies
pip install -r requirements.txt
-
Configure Tesseract Path (Windows only)
- Edit
app.pyline 41:pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract.exe'
- Edit
-
Start Ollama Service
ollama serve # Keep this running in a separate terminal -
Pull Gemma 2 Model
ollama pull gemma2:9b # ~5GB download, first time only -
Run Application
python app.py
-
Access Application
- Open browser:
http://localhost:5000
- Open browser:
Optional: Edit Configuration in app.py
app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024 # Max file size
app.config['MAX_CHAT_HISTORY'] = 100 # Max chats stored
app.config['CHAT_EXPIRY_DAYS'] = 7 # Days before chat expires
OLLAMA_MODEL = "gemma2:9b" # Change to "llama3" or "mistral" if preferredRate Limiting (Optional)
# Adjust in app.py
limiter = Limiter(
app=app,
key_func=get_remote_address,
default_limits=["200 per day", "50 per hour"] # Modify here
)| Issue | Solution |
|---|---|
| Tesseract not found | Ensure tesseract is in system PATH or update path in app.py |
| Ollama connection error | Run ollama serve in separate terminal before starting app |
| CUDA out of memory | Reduce image size or switch to CPU mode (set device = "cpu") |
| Model download fails | Check internet connection, manually pull: ollama pull gemma2:9b |
User uploads image via /upload endpoint
↓
Flask receives multipart/form-data
↓
Security validation (file type, size, extension)
↓
Generate secure filename (prevent path traversal)
↓
Save to uploads/ directory
↓
Generate unique chat_id (timestamp-based)
Load image with Pillow
↓
Resize to max 1024×1024 (LANCZOS resampling)
↓
Convert to RGB (if not already)
↓
Save optimized JPEG (quality=85)
↓
┌────────────────────────────────────────────┐
│ PARALLEL PROCESSING (3 threads) │
├────────────────────────────────────────────┤
│ Thread 1: Object Detection │
│ - Load optimized image │
│ - Convert to PyTorch tensor │
│ - Run Faster R-CNN inference │
│ - Filter detections (confidence > 0.5) │
│ - Map labels to COCO class names │
│ │
│ Thread 2: OCR Text Extraction │
│ - Load optimized image │
│ - Run Tesseract OCR │
│ - Extract all text regions │
│ - Return cleaned text │
│ │
│ Thread 3: Caption Generation │
│ - Collect detected objects + text │
│ - Build prompt for Ollama │
│ - Send request to Gemma 2:9B │
│ - Receive generated caption │
└────────────────────────────────────────────┘
↓
Aggregate results into JSON
↓
Store in processed_image_data[chat_id]
↓
Persist to chat_history.json
↓
Delete optimized image (cleanup)
↓
Encode original image to Base64
↓
Return JSON response to frontend
User asks question via /ask endpoint
↓
Retrieve cached image analysis from chat_id
↓
Build context prompt:
- Image caption
- Detected objects
- Extracted text
- User question
↓
Send to Ollama (Gemma 2:9B)
↓
LLM generates context-aware answer
↓
Store Q&A in chat history
↓
Update chat timestamp
↓
Save to chat_history.json
↓
Return answer to frontend
Cleanup thread runs every hour
↓
Iterate through chat_timestamps
↓
Identify chats older than 7 days
↓
Delete expired chats from:
- processed_image_data dict
- chat_timestamps dict
↓
Enforce max 100 chats limit:
- Sort chats by timestamp
- Remove oldest if exceeding limit
↓
Persist changes to chat_history.json
↓
Run garbage collection (gc.collect())
Faster R-CNN Inference:
# 1. Preprocessing
transform = transforms.Compose([transforms.ToTensor()])
img_tensor = transform(image).to(device) # Move to GPU/CPU
# 2. Forward pass (no gradient computation)
with torch.no_grad():
predictions = detection_model([img_tensor])
# 3. Post-processing
for score, label in zip(predictions[0]['scores'], predictions[0]['labels']):
if score > 0.5: # Confidence threshold
class_name = COCO_CLASSES[label.item()]
objects.append({'class': class_name, 'confidence': score.item()})Ollama LLM Inference:
response = ollama.chat(
model="gemma2:9b",
messages=[
{'role': 'system', 'content': 'You are an image analysis assistant'},
{'role': 'user', 'content': prompt}
],
options={
'temperature': 0.7, # Creativity level (0=deterministic, 1=creative)
'num_ctx': 4096 # Context window size
}
)Scenario: A visually impaired user needs to know what's in a photo sent by a friend.
Workflow:
- Upload image via screen reader-compatible interface
- Receive audio readout: "The image contains 2 people, a dog, and a car. Caption: A family enjoying a sunny day in the park with their golden retriever."
- Ask: "What color is the dog?"
- Response: "The dog appears to be golden in color, typical of a golden retriever breed."
Scenario: A blogger needs alt text for 50 images for SEO and accessibility.
Workflow:
- Batch upload images
- Extract captions: "A modern kitchen with stainless steel appliances and marble countertops"
- Copy captions directly to CMS alt-text fields
- Time saved: 30 seconds per image → 25 minutes total
Scenario: An accountant needs to digitize paper receipts for expense reports.
Workflow:
- Photograph receipt with phone
- Upload to app
- OCR extracts: "TOTAL: $45.99, DATE: 12/09/2025, MERCHANT: Coffee Shop"
- Copy text to accounting software
Scenario: A history teacher wants students to analyze a WWII photo.
Workflow:
- Upload historical photo
- System detects: military vehicles, soldiers, uniforms
- Students ask: "What type of vehicles are in the image?"
- LLM responds: "Based on the detected objects, these appear to be military trucks, possibly troop transports from the 1940s era."
Scenario: An online store needs to tag 1000 product images.
Workflow:
- Upload product photo
- Object detection identifies: chair, table, lamp
- Auto-generate tags for database
- Improves search accuracy and recommendation systems
Scenario: A security officer needs to document an incident with photos.
Workflow:
- Upload photo of damaged property
- System describes: "Image shows a broken window with shattered glass on the floor"
- Ask: "Are there any sharp objects visible?"
- Response: "Yes, there are visible glass shards on the floor which pose a safety hazard."
-
YOLO Integration
- Add YOLO as alternative detector for real-time video processing
- User toggle: Faster R-CNN (accuracy) vs. YOLO (speed)
-
Multi-Language OCR
- Support Hindi, Spanish, Chinese, Arabic
- Language auto-detection
-
Batch Upload
- Process multiple images simultaneously
- Bulk export to CSV/JSON
-
Advanced Captioning
- Image-to-image search (find similar images)
- Style transfer descriptions ("artistic," "photorealistic," etc.)
-
Database Integration
- Replace JSON with PostgreSQL/MongoDB
- Faster queries and scalability
-
User Authentication
- Login system with JWT tokens
- Private chat histories per user
-
API Endpoints
- RESTful API for third-party integrations
- Swagger documentation
-
Mobile App
- React Native or Flutter mobile client
- Camera integration for instant analysis
-
Fine-Tuned Models
- Train custom Faster R-CNN on domain-specific data (medical images, fashion, etc.)
- Fine-tune Gemma 2 for specialized captions
-
Video Analysis
- Frame-by-frame object tracking
- Generate video summaries
-
Cloud Deployment
- Dockerized deployment (Kubernetes)
- AWS/GCP hosting with auto-scaling
-
Advanced Analytics
- Dashboard showing popular objects, queries
- Heatmaps of object locations in images
- Load Balancing: Deploy multiple Flask workers (Gunicorn/uWSGI)
- Caching Layer: Redis for frequently accessed data
- CDN: Serve static assets via CDN (CloudFlare, AWS CloudFront)
- Async Processing: Celery task queue for background jobs
We welcome contributions! Here's how:
- Fork the repository
- Create a feature branch:
git checkout -b feature/your-feature - Commit changes:
git commit -m "Add your feature" - Push to branch:
git push origin feature/your-feature - Open a Pull Request
Contribution Guidelines:
- Follow PEP 8 style guide for Python
- Add unit tests for new features
- Update documentation (README, docstrings)
- Ensure all tests pass before submitting
This project is licensed under the MIT License.
MIT License
Copyright (c) 2025 Mithrajith
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
- GitHub: Mithrajith
- Issues: Report bugs or request features
- Discussions: Join community discussions
- PyTorch Team: For pre-trained Faster R-CNN models
- Google DeepMind: For open-sourcing Gemma 2
- Ollama: For simplifying local LLM deployment
- Tesseract OCR: For powerful text recognition
- Flask Community: For excellent web framework
Built with ❤️ for privacy-first, accessible AI