feat: productionize AI recommendation pipeline

[CoreyLeath-code]

Summary

Implements the repository-scoped production-readiness roadmap from #8 and replaces disconnected demos and unsupported claims with one verifiable training-to-serving path.

AI and evaluation

• adds versioned interaction contracts, temporal splits, causal training examples, deterministic training, and a popularity baseline
• reports Recall@K, NDCG@K, MRR@K, and catalogue coverage
• packages weights, vocabulary, architecture, lineage, metrics, and SHA-256 checksums in one immutable bundle
• exports the verified production architecture to ONNX

Serving and operations

• fixes item/output alignment and padding-aware attention
• fails closed in production when a verified trained bundle is unavailable
• adds optional API-key auth, rate limiting, concurrency admission, model-versioned cache, deterministic fallback, feedback capture, and Prometheus signals
• hardens Docker and Kubernetes execution with non-root/read-only settings, probes, and resource limits

Engineering quality and documentation

• removes mocked or disconnected recommendation paths
• makes CI, schema validation, security audit, coverage, benchmark, and container smoke checks fail honestly
• rewrites the README with accurate badges, L5 design decisions, reproducible measurements, limitations, engineering features, and extended recruiter Q&A
• adds a model card, benchmark protocol, operations runbook, and privacy notes

Validation

• 43 tests passed
• 91.71% coverage (85% gate)
• Ruff passed
• Python compilation passed
• Bandit reported no medium/high findings
• GitHub Actions and Kubernetes YAML parsed successfully
• deterministic CPU benchmark executed and emitted machine-readable results
• ONNX artifact passed checker validation
• diff hygiene passed

Known boundary

Docker Desktop was unavailable in the local execution environment. The PR adds a strict remote container build and trained-model health smoke job to validate that boundary in GitHub Actions. Demo data is explicitly pipeline-only; real ranking claims still require a production interaction snapshot and online experimentation.

Relates to #8.

Summary by CodeRabbit

• New Features

  • Added versioned, verified model bundles with checksum validation and model provenance.
  • Added production recommendation controls, including API-key protection, rate limiting, caching, concurrency limits, and latency-based fallback recommendations.
  • Added feedback submission with privacy-conscious logging.
  • Added training, evaluation, popularity-baseline comparison, and reproducible inference benchmarking.
  • Added production container and Kubernetes deployment support with readiness checks and hardened runtime settings.

• Documentation

  • Reworked setup, serving, architecture, operations, privacy, model card, and benchmarking guidance.

• Bug Fixes

  • Improved input validation, padding handling, item filtering, and model readiness reporting.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 912d6c32-10e9-4a68-8e6d-2c69d9a58579

📥 Commits

Reviewing files that changed from the base of the PR and between 010fe22 and fabfda0.

📒 Files selected for processing (57)

• .dockerignore
• .env.example
• .github/workflows/benchmarks.yml
• .github/workflows/ci-cd.yml
• .github/workflows/ci.yml
• .github/workflows/data-validation.yml
• .github/workflows/security.yml
• .gitignore
• Dockerfile
• README.md
• app/api/routes.py
• app/core/artifacts.py
• app/core/config.py
• app/core/data_processor.py
• app/core/metrics.py
• app/core/model.py
• app/core/security.py
• app/core/serving.py
• app/main.py
• app/recsys_control_room.py
• benchmarks/__init__.py
• benchmarks/inference.py
• demo_app.py
• docker-compose.yml
• docs/BENCHMARKS.md
• docs/MODEL_CARD.md
• docs/OPERATIONS.md
• docs/PRIVACY.md
• docs/architecture.md
• docs/demo_app.py
• docs/metrics.md
• k8s/deployment.yaml
• k8s/model-pvc.yaml
• pyproject.toml
• recommender/guards.py
• recommender/orchestrator.py
• recommender/state.py
• requirements-dev.txt
• requirements.txt
• scripts/__init__.py
• scripts/generate_demo_data.py
• src/features/online_retrieval.py
• src/serving/onnx_exporter.py
• src/serving/triton_config.pbtxt
• src/training/__init__.py
• src/training/baselines.py
• src/training/data.py
• src/training/metrics.py
• src/training/train.py
• streamlit_app.py
• tests/test_artifacts.py
• tests/test_benchmark.py
• tests/test_recommender.py
• tests/test_schema_contracts.py
• tests/test_serving_controls.py
• tests/test_smoke_api.py
• tests/test_training_pipeline.py

---

📝 Walkthrough

Walkthrough

The PR adds deterministic training and evaluation, checksummed model bundles, production-oriented recommendation serving, feedback handling, hardened containers and Kubernetes deployment, CI enforcement, benchmarks, and operational documentation.

Changes

Production recommender pipeline

Layer / File(s)	Summary
Training data and evaluation pipeline src/training/*, scripts/*, tests/test_schema_contracts.py, tests/test_training_pipeline.py, pyproject.toml	Adds versioned event processing, temporal splits, ranking metrics, popularity baselines, deterministic training, model promotion reports, demo-data generation, and end-to-end validation.
Model contract and verified artifacts app/core/model.py, app/core/artifacts.py, app/core/data_processor.py, src/serving/onnx_exporter.py, tests/test_artifacts.py, tests/test_recommender.py	Adds padding-aware ranking, vocabulary persistence, checksummed model bundles, strict bundle loading, bundle-based ONNX export, and integrity tests.
Runtime initialization and serving controls app/main.py, app/api/routes.py, app/core/{config,security,serving,metrics}.py, streamlit_app.py, tests/test_serving_controls.py, tests/test_smoke_api.py	Adds verified startup loading, API-key authorization, rate limiting, caching, admission control, latency and overload fallbacks, feedback logging, readiness details, and model provenance.
Container, deployment, and CI enforcement Dockerfile, docker-compose.yml, k8s/*, .github/workflows/*, requirements*.txt, .env.example, .dockerignore, .gitignore	Updates multi-stage container builds, non-root/read-only runtime settings, model-bundle storage and probes, dependency tooling, coverage and security gates, benchmark artifacts, and container smoke testing.
Benchmarking and operational documentation benchmarks/*, docs/*, README.md	Adds deterministic inference benchmarks and documents artifact promotion, operations, privacy, architecture, metrics, production serving, and project usage.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

• Issue #8 — The PR implements the issue’s objectives around trained bundle loading, model/indexing validation, reproducible benchmarks, production serving controls, security, and CI enforcement.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant FastAPI
  participant RecommendationCache
  participant DeepSequenceModel
  Client->>FastAPI: Send recommendation request
  FastAPI->>RecommendationCache: Check versioned cache
  RecommendationCache-->>FastAPI: Return hit or miss
  FastAPI->>DeepSequenceModel: Run bounded inference
  DeepSequenceModel-->>FastAPI: Return ranked recommendations
  FastAPI-->>Client: Return recommendations and provenance

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/production-ai-readiness

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Checkov (3.3.8)

.github/workflows/benchmarks.yml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

.github/workflows/ci-cd.yml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

.github/workflows/ci.yml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

• 7 others

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}