Synthscholar integration

Dockerfile.unified

@@ -5,7 +5,8 @@
 # Oxigraph will be run as a separate service in docker-compose or use the fallback script
 
 # Main build stage
-FROM python:3.10-slim
+# Python 3.11+ required: synthscholar (ml_service) depends on it.
+FROM python:3.11-slim
 
 # Set metadata
 LABEL project="BrainyPedia" \
@@ -46,7 +47,7 @@ COPY ml_service/ /app/ml_service/
 WORKDIR /app/ml_service
 RUN pip install --use-deprecated=legacy-resolver "structsense==0.0.4" || \
     pip install --use-deprecated=legacy-resolver --no-deps "structsense==0.0.4" && \
-    pip install -r requirements.txt
+    pip install --use-deprecated=legacy-resolver -r requirements.txt
 
 # Copy usermanagement_service
 COPY usermanagement_service/ /app/usermanagement_service/

README.unified-docker.md

@@ -4,11 +4,16 @@ This Dockerfile deploys the following BrainKB backend services:
 - **APItokenmanager** (Django) - Port 8000
 - **query_service** (FastAPI) - Port 8010
 - **ml_service** (FastAPI) - Port 8007
+  - Hosts the **SynthScholar** PRISMA literature-review pipeline at
+    `/api/synth-scholar/*` alongside the existing structsense routes.
+    SynthScholar reuses the unified `brainkb` Postgres database (no extra
+    DSN config required); see the SynthScholar block in `env.template` for
+    the optional API keys it consumes (OpenRouter / NCBI / Semantic Scholar
+    / CORE).
 - **usermanagement_service** (FastAPI) - Port 8004
 - **oxigraph** (SPARQL Database) - Port 7878
 
 **Services NOT included in this unified deployment (deploy separately):**
-- **brainkb-ui** - The UI is not included. See [SETUP_UI.md](SETUP_UI.md) for UI deployment instructions.
 - **chat_service** - Deploy separately using `chat_service/docker-compose-prod.yml` or `chat_service/docker-compose-dev.yml`.
 
 ## Quick Start with Docker Compose (Recommended)
@@ -549,6 +554,7 @@ All environment variables are loaded from the `.env` file in the project root. T
 - **User Management OAuth**: `USERMANAGEMENT_SERVICE_JWT_SECRET_KEY`, `USERMANAGEMENT_PUBLIC_BASE_URL`, `USERMANAGEMENT_FRONTEND_CALLBACK_URL`, `USERMANAGEMENT_OAUTH_TOKEN_ENC_KEY`, `USERMANAGEMENT_BOOTSTRAP_SUPERADMIN_EMAILS`, `GITHUB_CLIENT_ID/SECRET`, `ORCID_CLIENT_ID/SECRET`, `GLOBUS_CLIENT_ID/SECRET`
 - **Ollama**: `OLLAMA_MODEL`, `OLLAMA_PORT`, `OLLAMA_API_ENDPOINT`
 - **ML Service**: `MONGO_DB_URL`, `WEAVIATE_*`, etc.
+- **SynthScholar** (PRISMA reviews, lives in ml_service): `OPENROUTER_API_KEY` (operator fallback — UI normally forwards a per-user or admin-shared key), `NCBI_API_KEY` (optional, raises PubMed rate limits), `SEMANTIC_SCHOLAR_API_KEY` / `CORE_API_KEY` / `SYNTHSCHOLAR_EMAIL` (all optional). Database is shared — no separate DSN.
 - **Query Service**: `GRAPHDATABASE_*`, `RAPID_RELEASE_FILE`
 
 See `env.template` for the complete list of all available environment variables with descriptions.

docker-compose.unified.yml

@@ -163,4 +163,9 @@ volumes:
 networks:
   brainkb-network:
     external: true
+    # Pin the network name so it stays the same regardless of the compose
+    # project (folder) name. The brainkb-ui stack joins the same network by
+    # this fixed name. Create it once before bringing up either stack:
+    #     docker network create brainkb-network
+    name: brainkb-network
 

env.template

@@ -123,6 +123,8 @@ PGADMIN_PORT=5051
 # ----------------------------------------------------------------------------
 OXIGRAPH_USER=admin
 OXIGRAPH_PASSWORD=your_oxigraph_password_change_this
+OXIGRAPH_DATA_PATH=/
+OXIGRAPH_TMP_PATH=/
 
 # Oxigraph Data Storage Configuration
 # For local development: Leave commented to use Docker named volumes (recommended)
@@ -194,6 +196,55 @@ OLLAMA_API_ENDPOINT=http://host.docker.internal:11434
 OLLAMA_MODEL=nomic-embed-text
 OLLAMA_PORT=11434
 
+# ----------------------------------------------------------------------------
+# ML Service - SynthScholar (PRISMA literature review)
+# ----------------------------------------------------------------------------
+# Mounts under /api/synth-scholar/* in ml_service. End-to-end PRISMA-guided
+# pipeline (search strategy → article fetch → title/abstract + full-text
+# screening → data charting → critical appraisal → narrative synthesis →
+# GRADE), driven by the `synthscholar` Python library and orchestrated by
+# the local module at ml_service/core/synth_scholar/. SSE progress, plan-
+# confirmation gating, exports (markdown / JSON / BibTeX / Turtle / JSON-LD),
+# per-user ownership, and shared/cached corpus search are all wired up.
+#
+# Database: SynthScholar reuses the unified `brainkb` Postgres database
+# (the same one JWT auth and usermanagement_service write to). Its tables
+# (`reviews`, plus synthscholar's bundled `article_store` / `review_cache` /
+# `pipeline_checkpoints`) coexist with the existing `Web_*` tables — names
+# are snake_case here vs PascalCase there, so there are no collisions. No
+# separate DSN is needed; the JWT_POSTGRES_DATABASE_* values above are
+# reused at startup. Optional: install pgvector in the postgres image to
+# enable semantic article/review search; without it, only keyword/title
+# search works (synthscholar's bundled migration 004 logs and skips).
+#
+# Auth: every endpoint sits behind ml_service's existing JWT (clients call
+# /api/token with NEXT_PUBLIC_JWT_USER / _PASSWORD on the UI side). The
+# UI also forwards a per-user OpenRouter key on every create-review call
+# (resolved from the user's personal sessionStorage key, or the admin-
+# shared key set in /admin/settings/openrouter-key). The OPENROUTER_API_KEY
+# below is only the operator fallback used when neither is configured.
+
+# OpenRouter API key — operator fallback only. The UI normally forwards a
+# per-user or admin-shared key on every request, so this env value is used
+# only when neither is configured. Safe to leave blank in production.
+OPENROUTER_API_KEY=
+
+# Optional. NCBI E-utilities API key — recommended in production to raise
+# PubMed rate limits from 3 to 10 requests/second.
+# https://www.ncbi.nlm.nih.gov/account/settings/
+NCBI_API_KEY=
+
+# Optional. Semantic Scholar API key — adds Semantic Scholar to the search
+# fan-out. Silently skipped at runtime if absent.
+SEMANTIC_SCHOLAR_API_KEY=
+
+# Optional. CORE API key — adds the CORE provider. Silently skipped if absent.
+CORE_API_KEY=
+
+# Optional. Contact email passed to OpenAlex / Crossref / Europe PMC for
+# polite-pool rate limits. Silently skipped if absent.
+SYNTHSCHOLAR_EMAIL=
+
 
 
 

ml_service/Dockerfile

@@ -1,4 +1,4 @@
-FROM tiangolo/uvicorn-gunicorn-fastapi:python3.10
+FROM tiangolo/uvicorn-gunicorn-fastapi:python3.11
 
 # Set metadata for the image
 LABEL project="BrainyPedia" \

ml_service/core/main.py

@@ -13,12 +13,24 @@
 
 from core.configure_logging import configure_logging
 from core.routers.index import router as index_router
-from core.routers.jwt_auth import router as jwt_router 
+from core.routers.jwt_auth import router as jwt_router
 from core.routers.structsense import router as structsense_router
 from core.database import init_db_pool, get_db_pool, debug_pool_status
 from core.configuration import load_environment
 from motor.motor_asyncio import AsyncIOMotorClient
 
+# SynthScholar (PRISMA literature review). Imports are lazy-guarded so a
+# missing `synthscholar` library doesn't crash the rest of ml_service —
+# the router simply won't mount and the /api/synth-scholar/* surface returns 404.
+try:
+    from core.synth_scholar.routes import router as synth_scholar_router
+    from core.synth_scholar.database import init_db as init_synth_scholar_db, close_db as close_synth_scholar_db
+    from core.synth_scholar.store import fix_stuck_reviews as fix_synth_scholar_stuck_reviews
+    _SYNTH_SCHOLAR_AVAILABLE = True
+except ImportError as _exc:
+    _SYNTH_SCHOLAR_AVAILABLE = False
+    _SYNTH_SCHOLAR_IMPORT_ERROR = _exc
+
 from fastapi.middleware.cors import CORSMiddleware
 
 # Initialize logger - will be configured in lifespan
@@ -44,6 +56,25 @@ async def lifespan(app: FastAPI):
         logger.error(f"Failed to initialize database pool: {e}")
         raise
 
+    # Initialise SynthScholar (PRISMA review) tables and recover any sessions
+    # that were running when the server last shut down. Failures here are
+    # logged but non-fatal so the rest of ml_service still boots.
+    if _SYNTH_SCHOLAR_AVAILABLE:
+        try:
+            await init_synth_scholar_db()
+            stuck = await fix_synth_scholar_stuck_reviews()
+            if stuck:
+                logger.warning("SynthScholar: marked %d stuck review(s) as FAILED on startup", stuck)
+            logger.info("SynthScholar: database ready")
+        except Exception as e:
+            logger.error("SynthScholar startup failed (continuing without it): %s", e)
+    else:
+        logger.warning(
+            "SynthScholar disabled — import failed: %s. Install `synthscholar` "
+            "and `sqlalchemy[asyncio]` to enable.",
+            _SYNTH_SCHOLAR_IMPORT_ERROR,
+        )
+
     # Initialize MongoDB client (reused across all requests)
     try:
         env = load_environment()
@@ -113,6 +144,14 @@ async def lifespan(app: FastAPI):
     except Exception as e:
         logger.error(f"Error closing MongoDB client: {e}")
 
+    # Dispose the SynthScholar SQLAlchemy engine.
+    if _SYNTH_SCHOLAR_AVAILABLE:
+        try:
+            await close_synth_scholar_db()
+            logger.info("SynthScholar: SQLAlchemy engine disposed")
+        except Exception as e:
+            logger.error("SynthScholar shutdown error: %s", e)
+
     logger.info("FastAPI shutdown complete")
 
 app = FastAPI(
@@ -150,6 +189,8 @@ async def lifespan(app: FastAPI):
 app.include_router(index_router, prefix="/api")
 app.include_router(jwt_router, prefix="/api", tags=["Security"])
 app.include_router(structsense_router, prefix="/api", tags=["Multi-agent Systems"])
+if _SYNTH_SCHOLAR_AVAILABLE:
+    app.include_router(synth_scholar_router, prefix="/api")
 
 # Exception handlers
 @app.exception_handler(HTTPException)

ml_service/core/synth_scholar/__init__.py

@@ -0,0 +1,10 @@
+"""SynthScholar — PRISMA-guided literature review pipeline.
+
+Ported from aep-knowledge-synthesis/backend. Uses an independent SQLAlchemy
+2.0 async engine for its own tables (reviews, plus synthscholar's bundled
+article_store / review_cache / pipeline_checkpoints), separate from the
+existing ml_service raw-asyncpg pool used by jwt_auth + structsense.
+
+Entry point: `core.synth_scholar.routes.router` (mounted at /api/synth-scholar
+by core/main.py). All endpoints require a valid JWT (Depends(get_current_user)).
+"""