🤖 Intelligent Query - AI-Powered PDF Q&A System

A sophisticated AI-powered document analysis system that enables users to upload PDF documents and ask intelligent questions about their content. Built with modern AI technologies including Groq's ultra-fast inference, semantic search, and advanced document processing.

🌟 Key Features

� Advanced Document Processing

Multi-format Support: PDF, DOCX, and email files (.eml)
Large File Handling: Support for documents up to 200MB
Fast Extraction: PyMuPDF with pdfplumber fallback for optimal performance
Smart Chunking: Intelligent text segmentation preserving context

🧠 AI-Powered Intelligence

Groq Integration: Ultra-fast inference with Llama 3.1 8B Instant model
Semantic Search: FAISS-powered vector similarity search
Context-Aware Responses: Maintains document context across conversations
Multiple AI Models: Support for various embedding models

🚀 Performance & Scalability

Intelligent Caching: Document and model caching with TTL
Async Processing: Concurrent request handling
Rate Limiting: Built-in protection against abuse
Memory Optimization: Efficient resource management

🔒 Enterprise-Ready Security

Bearer Token Authentication: Secure API access
Input Validation: Comprehensive security checks
Non-root Execution: Docker security best practices
Environment Isolation: Secure configuration management

🌐 Dual Interface Options

Web Interface: Modern, responsive chat-based UI
REST API: Full programmatic access with OpenAPI documentation
Docker Support: Containerized deployment ready

📚 Documentation

Complete documentation is available in the docs/ directory:

Project Structure - Detailed module and directory organization
Developer Guide - For developers setting up the project
Deployment Guide - Production deployment instructions
User Guide - End-user documentation
API Documentation - Complete API reference
Docker Deployment - Docker setup and management
Docker Hub Guide - Docker Hub integration
Architecture Improvements - Refactoring roadmap
Phase 1 Complete - Architecture foundation summary
Codebase Organization - Current cleanup status
Quick Reference - Developer quick reference

🏗️ Architecture Overview

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Web Interface │    │    REST API      │    │   Docker        │
│   (Flask)       │    │   (FastAPI)      │    │   Container     │
└─────────┬───────┘    └────────┬─────────┘    └─────────┬───────┘
          │                     │                        │
          └─────────────────────┼────────────────────────┘
                                │
                    ┌───────────▼───────────┐
                    │   Core Engine         │
                    │   - Document Parser   │
                    │   - Embedding Engine  │
                    │   - Vector Search     │
                    │   - AI Integration    │
                    └───────────┬───────────┘
                                │
          ┌─────────────────────┼─────────────────────┐
          │                     │                     │
    ┌─────▼─────┐    ┌─────────▼────────┐    ┌──────▼──────┐
    │  PyMuPDF  │    │ SentenceTransf.  │    │   Groq API  │
    │ PDFPlumber│    │     FAISS        │    │ Llama 3.1   │
    └───────────┘    └──────────────────┘    └─────────────┘

🚀 Quick Start

Prerequisites

Python 3.11 or higher
Git
Groq API key (Get one here)

1. Clone the Repository

git clone https://github.com/your-username/intelligent-query.git
cd intelligent-query

2. Set Up Virtual Environment

# Create virtual environment
python -m venv venv

# Activate virtual environment
# Windows:
.\venv\Scripts\activate
# macOS/Linux:
source venv/bin/activate

3. Install Dependencies

pip install --upgrade pip
pip install -r requirements.txt

4. Configure Environment

# Copy environment template
cp .env.example .env

# Edit .env file and add your Groq API key
# GROQ_API_KEY=your_groq_api_key_here

5. Run the Application

Option A: Web Interface (Recommended for beginners)

python run_flask.py

Open your browser and go to: http://localhost:5000

Option B: API Server (For developers)

python -c "import uvicorn; uvicorn.run('src.app:app', host='0.0.0.0', port=3000)"

API Documentation: http://localhost:3000/docs

6. Test the Setup

python test_setup.py

📖 Usage Guide

Web Interface Usage

Upload Document: Drag and drop a PDF file or click to browse
Wait for Processing: The system will extract and index the document
Ask Questions: Type your questions in the chat interface
Get Answers: Receive AI-powered responses based on document content

API Usage

Authentication

All API requests require a Bearer token:

curl -H "Authorization: Bearer YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     http://localhost:3000/health

Process Document and Ask Questions

curl -X POST "http://localhost:3000/hackrx/run" \
     -H "Authorization: Bearer YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "documents": "https://example.com/document.pdf",
       "questions": [
         "What is the main topic of this document?",
         "What are the key findings?"
       ]
     }'

Response Format

{
  "answers": [
    "The main topic of this document is artificial intelligence and machine learning applications in healthcare.",
    "The key findings include improved diagnostic accuracy and reduced processing time."
  ]
}

🐳 Docker Deployment

Quick Docker Setup

# Build and run with Docker Compose
docker-compose up --build -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f

Manual Docker Commands

# Build image
docker build -t intelligent-query .

# Run container
docker run -d \
  --name intelligent-query-app \
  -p 5000:5000 \
  --env-file .env \
  -v $(pwd)/uploads:/app/uploads \
  intelligent-query

🔧 Configuration

Environment Variables

Variable	Description	Default	Required
`GROQ_API_KEY`	Groq API key for AI inference	-	✅
`SECRET_KEY`	Flask secret key	Generated	✅
`PORT`	Application port	3000	❌
`MAX_FILE_SIZE`	Maximum upload size (bytes)	16777216	❌
`DEBUG`	Enable debug mode	False	❌
`HACKRX_BEARER_TOKEN`	API authentication token	-	✅

Advanced Configuration

Model Selection

# In src/app.py, modify get_sentence_transformer()
model = SentenceTransformer('all-MiniLM-L6-v2')  # Fast
model = SentenceTransformer('all-mpnet-base-v2')  # Balanced
model = SentenceTransformer('BAAI/bge-large-en-v1.5')  # Accurate

Cache Settings

# Document cache configuration
MAX_CACHE_SIZE = 10  # Number of documents to cache
CACHE_TTL = 3600     # Cache time-to-live in seconds

📊 Performance Benchmarks

Processing Speed

Document Processing: ~30-60 seconds for 100-page PDF
Question Answering: ~2-5 seconds per question
Concurrent Users: Supports 10+ simultaneous users

Resource Usage

Memory: ~2-4GB RAM for optimal performance
Storage: ~1GB for models and cache
CPU: Multi-core recommended for concurrent processing

Optimization Tips

Use SSD storage for faster model loading
Increase RAM for larger document caches
Enable GPU for faster embedding generation (optional)
Use CDN for static assets in production

🧪 Testing

Run All Tests

# Environment setup test
python test_setup.py

# API client test
python test_openai_client.py

# Import tests
python test_web_app_imports.py

# Performance benchmark
python test/benchmark.py

API Testing

# Test health endpoint
curl http://localhost:3000/health

# Test with sample document
python scripts/test_openrouter.py

🔍 Troubleshooting

Common Issues

1. Import Errors

Problem: ModuleNotFoundError or import issues Solution:

# Ensure virtual environment is activated
.\venv\Scripts\activate  # Windows
source venv/bin/activate  # macOS/Linux

# Reinstall dependencies
pip install -r requirements.txt

2. API Key Issues

Problem: "API key not configured" error Solution:

# Check .env file exists and contains valid key
cat .env | grep GROQ_API_KEY

# Test API key
python test_openai_client.py

3. Memory Issues

Problem: Out of memory errors Solution:

Reduce MAX_CACHE_SIZE in configuration
Use smaller embedding models
Process smaller documents
Increase system RAM

4. Port Conflicts

Problem: "Port already in use" error Solution:

# Find process using port
netstat -ano | findstr :5000  # Windows
lsof -i :5000                 # macOS/Linux

# Kill process or change port in .env

Debug Mode

Enable debug logging:

# Set environment variable
export DEBUG=True  # Linux/macOS
set DEBUG=True     # Windows

# Or modify .env file
DEBUG=True

🛠️ Development

Project Structure

intelligent-query/
├── src/                    # Source code
│   ├── app.py             # FastAPI application
│   ├── web_app.py         # Flask web interface
│   └── new_app.py         # Optimized FastAPI version
├── scripts/               # Utility scripts
│   ├── setup-docker.bat   # Docker setup (Windows)
│   └── test_*.py          # Test scripts
├── docs/                  # Documentation
├── test/                  # Test files
├── uploads/               # File upload directory
├── requirements.txt       # Python dependencies
├── Dockerfile            # Docker configuration
├── docker-compose.yml    # Multi-container setup
└── .env.example          # Environment template

Adding New Features

1. New Document Format Support

# In src/app.py, add new extraction function
def extract_text_from_new_format(file_path):
    # Implementation here
    pass

# Update download_and_extract_text function
elif ext in ['.new_ext']:
    return extract_text_from_new_format(tmp_path)

2. Custom AI Models

# Add new model in get_sentence_transformer()
def get_custom_model():
    return SentenceTransformer('your-custom-model')

3. New API Endpoints

# In src/app.py
@app.post("/new-endpoint")
async def new_endpoint():
    # Implementation here
    pass

Code Style Guidelines

Follow PEP 8 for Python code
Use type hints where possible
Add docstrings for functions
Keep functions focused and small
Use meaningful variable names

🚀 Deployment

Production Deployment

1. Environment Setup

# Production environment variables
FLASK_ENV=production
DEBUG=False
SECRET_KEY=your_strong_secret_key
GROQ_API_KEY=your_api_key

2. Docker Production

# docker-compose.prod.yml
services:
  app:
    build: .
    restart: always
    ports:
      - "80:5000"
    environment:
      - FLASK_ENV=production
    volumes:
      - ./uploads:/app/uploads

3. Cloud Deployment Options

Railway.app:

# Install Railway CLI
npm install -g @railway/cli

# Deploy
railway login
railway init
railway up

Heroku:

# Create Heroku app
heroku create your-app-name

# Set environment variables
heroku config:set GROQ_API_KEY=your_key

# Deploy
git push heroku main

AWS/GCP/Azure:

Use Docker container deployment
Configure load balancer for scaling
Set up persistent storage for uploads
Configure environment variables securely

Scaling Considerations

Load Balancing: Use nginx or cloud load balancers
Database: Consider PostgreSQL for persistent storage
Caching: Redis for distributed caching
Monitoring: Implement health checks and logging
Security: HTTPS, rate limiting, input validation

📈 Monitoring & Analytics

Health Monitoring

# Check application health
curl http://localhost:3000/health

# Monitor Docker containers
docker-compose ps
docker-compose logs -f

Performance Metrics

Response time per request
Document processing time
Cache hit/miss ratios
Memory and CPU usage
API error rates

Logging

Logs are written to:

Console output
app.log (FastAPI)
web_app.log (Flask)

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Development Setup

Fork the repository
Create a feature branch
Make your changes
Add tests
Submit a pull request

Reporting Issues

Please include:

Environment details (OS, Python version)
Steps to reproduce
Expected vs actual behavior
Error messages and logs

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Groq for ultra-fast AI inference
Hugging Face for transformer models
FAISS for efficient similarity search
PyMuPDF for fast PDF processing
FastAPI and Flask for web frameworks

📚 Documentation

Complete Documentation Suite

📖 User Guide - Complete user manual with best practices
🔌 API Documentation - Comprehensive API reference
🚀 Deployment Guide - Deploy to any platform
🛠️ Developer Guide - Architecture and development
🤝 Contributing Guidelines - How to contribute
📝 Changelog - Version history and updates

Quick Links

🎯 Getting Started - Set up in 5 minutes
💡 Usage Examples - Common use cases
🐳 Docker Deployment - Containerized setup
🔧 Configuration - Environment setup
🧪 Testing - Quality assurance

📞 Support

📖 Documentation: Complete Documentation Suite
🐛 Issues: GitHub Issues
💬 Discussions: GitHub Discussions
📧 Email: support@your-domain.com

🗺️ Roadmap

Version 2.0 (Planned)

Version 2.1 (Future)

Multi-modal analysis (text + images)
Voice interaction
Mobile app
Enterprise SSO integration

Made with ❤️ by the Intelligent Query Team

Transform your documents into intelligent conversations!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

🤖 Intelligent Query - AI-Powered PDF Q&A System

🌟 Key Features

� Advanced Document Processing

🧠 AI-Powered Intelligence

🚀 Performance & Scalability

🔒 Enterprise-Ready Security

🌐 Dual Interface Options

📚 Documentation

🏗️ Architecture Overview

🚀 Quick Start

Prerequisites

1. Clone the Repository

2. Set Up Virtual Environment

3. Install Dependencies

4. Configure Environment

5. Run the Application

Option A: Web Interface (Recommended for beginners)

Option B: API Server (For developers)

6. Test the Setup

📖 Usage Guide

Web Interface Usage

API Usage

Authentication

Process Document and Ask Questions

Response Format

🐳 Docker Deployment

Quick Docker Setup

Manual Docker Commands

🔧 Configuration

Environment Variables

Advanced Configuration

Model Selection

Cache Settings

📊 Performance Benchmarks

Processing Speed

Resource Usage

Optimization Tips

🧪 Testing

Run All Tests

API Testing

🔍 Troubleshooting

Common Issues

1. Import Errors

2. API Key Issues

3. Memory Issues

4. Port Conflicts

Debug Mode

🛠️ Development

Project Structure

Adding New Features

1. New Document Format Support

2. Custom AI Models

3. New API Endpoints

Code Style Guidelines

🚀 Deployment

Production Deployment

1. Environment Setup

2. Docker Production

3. Cloud Deployment Options

Scaling Considerations

📈 Monitoring & Analytics

Health Monitoring

Performance Metrics

Logging

🤝 Contributing

Development Setup

Reporting Issues

📄 License

🙏 Acknowledgments

📚 Documentation

Complete Documentation Suite

Quick Links

📞 Support

🗺️ Roadmap