feature: Add recover stale jobs

[varit05]

Summary by CodeRabbit

• New Features

  • Added improved retrieval options, including reranking, diverse results, and multi-query expansion.
  • Added support for batch query embeddings and a new retrieval pipeline for search and Q&A.
  • Expanded configuration options for retrieval behavior and UI request timeout.

• Bug Fixes

  • Search and summary requests now return a consistent service-unavailable error when backend connectivity fails.
  • Ingestion now skips content that is already fully indexed and better handles recovery after interrupted jobs.
  • Startup is more resilient when SSL certificate files are missing.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2bd3745d-6323-4e0b-bbab-584378bcc445

📥 Commits

Reviewing files that changed from the base of the PR and between 9b58e19 and 09822db.

📒 Files selected for processing (12)

• .env.example
• .gitignore
• api/main.py
• api/routes/ingest.py
• api/routes/query.py
• config.py
• processing/embedder.py
• rag/__init__.py
• rag/pipeline.py
• rag/qa_chain.py
• rag/retriever.py
• rag/search.py

---

📝 Walkthrough

Walkthrough

Adds a new RAGPipeline module combining multi-query expansion, cross-encoder reranking, and MMR diversity selection, wires it into QAChain and SearchEngine, revises HyDE and system prompts, and adds corresponding config/env settings. Also adds API robustness: stale ingest job recovery, conditional SSL, ingestion idempotency, and 503 error handling.

Changes

RAG Pipeline Overhaul

Layer / File(s)	Summary
Configuration and environment settings .env.example, config.py	Adds reranker, MMR, multi-query, and HyDE settings with defaults, adds ui_request_timeout, and changes whisper_model_size default to "base".
Batch query embedding processing/embedder.py	Adds embed_query_batch to embed multiple query strings, routing to local or OpenAI embedding.
RAGPipeline implementation rag/pipeline.py, rag/__init__.py	New RAGPipeline class orchestrates multi-query expansion, retrieval, deduplication, cross-encoder reranking, and MMR diversity selection; exported from package init.
Wire into qa_chain and search rag/qa_chain.py, rag/search.py, rag/retriever.py	QAChain and SearchEngine now use RAGPipeline for retrieval; system prompts, HyDE prompts, and context formatting are revised.

Estimated code review effort: 4 (Complex) | ~50 minutes

API Robustness Fixes

Layer / File(s)	Summary
Startup stale job recovery and conditional SSL api/main.py	Adds _recover_stale_jobs() invoked at startup to mark stuck ingest jobs as errored; Uvicorn SSL is enabled only when cert/key files exist.
Ingestion idempotency and error cleanup api/routes/ingest.py	Skips re-processing already-indexed videos; replaces silent exception handling with rollback and warning logs.
503 error handling for LLM failures api/routes/query.py	Wraps chain.ask and summarizer.summarize calls to raise HTTPException(503) on failure, with updated OpenAPI response docs.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant QAChain
  participant RAGPipeline
  participant VectorStore
  participant CrossEncoder
  QAChain->>RAGPipeline: retrieve(query, top_k, video_id)
  RAGPipeline->>RAGPipeline: generate query variants
  RAGPipeline->>VectorStore: retrieve per query variant
  VectorStore-->>RAGPipeline: candidate chunks
  RAGPipeline->>CrossEncoder: rerank candidates
  CrossEncoder-->>RAGPipeline: reranked scores
  RAGPipeline->>RAGPipeline: MMR diversity selection
  RAGPipeline-->>QAChain: top_k SearchResults

Loading

Possibly related PRs

• varit05/Streaming-Video-RAG#3: Both PRs change how rag/qa_chain.py and rag/search.py retrieve chunks, switching between retriever implementations.
• varit05/Streaming-Video-RAG#6: Both PRs touch config.py's whisper_model_size default and related rag/qa_chain.py/rag/retriever.py code.
• varit05/Streaming-Video-RAG#15: Both PRs modify HyDE prompt constants in rag/retriever.py along with corresponding HyDE settings in .env.example/config.py.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/add-recovering-stale-jobs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}