VIGIL — Vision-Informed Guardian for Infant Life

An offline, AI-powered neonatal danger-sign assessment tool for community health workers in low-resource settings. Built for The Gemma 4 Good Hackathon (2026).

VIGIL turns a laptop (or an Android phone — see the separate VIGIL Mobile repo) into a portable WHO-IMNCI neonatal assessment station. A trained CHW follows a guided voice + camera workflow; a fine-tuned Gemma 4 E4B runs a three-pass multimodal reasoning chain entirely on-device; deterministic WHO-derived tools compute every dose, z-score and ORS volume; and the result is a colour-coded danger-sign classification with an auditable record. No internet required after setup.

This repository is the source of truth for the model: the laptop application, the deployed clinical ML heads, the trained VIGIL LoRA adapter, the training/evaluation scripts, and the held-out evaluation reports.

Licensed under CC BY 4.0 — see LICENSE. VIGIL is clinical decision support for trained CHWs; outputs must be reviewed by a qualified health worker. It does not replace clinician judgment or local emergency protocols.

---

Features

Voice intake	The CHW speaks the mother's concerns in their local language; Whisper.cpp transcribes offline, Gemma 4 interprets the clinical content.
7-step guided camera capture	On-screen positioning overlays for face, eyes, chest, umbilicus, full body, foot sole (gestational age), and optional placenta. Flashlight toggle and file-upload fallback for the field.
Respiratory & cry assessment	Tap-to-count breathing-rate counter with median filtering (flags fast breathing at ≥ 60/min); 10-second cry recording.
Three-pass Gemma 4 reasoning	Pass 1 — 13 visual danger-sign checks from the images; Pass 2 — 8 audio / verbal / reported checks; Pass 3 — synthesis into a WHO IMNCI classification + tool calls. 21 strictly-templated, machine-parseable lines, streamed live over SSE.
Hybrid LLM + ML decision layer	Two calibrated LightGBM heads are ensembled with Gemma 4 — one distils the WHO IMNCI danger-sign algorithm, one grades neonatal jaundice colorimetrically against the printed colour-reference card in the frame.
5 deterministic clinical tools	Invoked via Gemma 4's native function calling, but computed in hard-coded WHO-derived code: weight-for-age z-score (WHO 2006 LMS), medication dosing with hard safety caps, ORS volume, referral letter, follow-up date.
WHO traffic-light result	🔴 urgent referral / 🟡 treatment needed / 🟢 home care, on top of a 21-check audit grid; Piper TTS speaks the action plan back in the CHW's language.
PSBI "Referral Not Possible" flow	When hospital referral is physically impossible, switches to the WHO 2015 simplified outpatient antibiotic regimen with hard-capped doses.
Training Mode	The same Gemma 4 becomes an offline CHW skill trainer — curated visual / audio / IMNCI-classification drills, deterministic grading, local progress tracking, next-case recommendation based on past mistakes, and an optional Gemma coach.
12 languages + RTL	English, French, Swahili, Hausa, Hindi, Bengali, Arabic, Portuguese, Spanish, Amharic, Indonesian, Urdu (RTL for Arabic and Urdu).
Encounter persistence	Every assessment saved locally as a structured JSON record (images, 21-check grid, classification, tool outputs) plus a client-side printable report for supervisor review. No cloud sync.
100% offline	Model, speech-to-text (Whisper.cpp), text-to-speech (Piper), backend and frontend all run locally. GPU or CPU inference.

Architecture

┌──────────────────────┐   ┌──────────────────────┐   ┌──────────────────────┐
│  Frontend  :3000     │──▶│  FastAPI backend :8000│──▶│  llama-server :8080  │
│  Vite + TypeScript   │   │  3-pass orchestrator  │   │  Gemma 4 E4B (IQ4_XS)│
│  Material Design 3    │   │  deterministic tools  │   │  + mmproj-BF16 vision│
│  12-lang i18n + RTL   │   │  LightGBM ML ensemble │   │  + VIGIL GGUF LoRA   │
│  camera overlays      │   │  JSON encounter store │   │  GPU or CPU           │
└──────────────────────┘   └──────────┬───────────┘   └──────────────────────┘
                                       │ proxies to optional offline sidecars
                              ┌────────┴────────┐   ┌─────────────────────┐
                              │ Whisper.cpp :8178│   │ Piper TTS :5000      │
                              │  speech → text   │   │  text → speech       │
                              └──────────────────┘   └─────────────────────┘

• Inference engine — llama.cpp. Gemma 4 E4B-IT quantised to IQ4_XS (~4 GB) is served through an OpenAI-compatible streaming endpoint; the multimodal projector (mmproj-BF16.gguf) enables native image understanding; the trained VIGIL LoRA is applied as a llama.cpp-native GGUF adapter at load time. --n-gpu-layers 0 runs the whole model in system RAM — inference on an 8 GB-RAM laptop with no GPU at all.
• Safety-critical determinism. All medication dosing, z-scores and ORS volumes are computed by hard-coded WHO-derived functions. The model triggers tool calls; the functions enforce the non-negotiable caps. The model never writes a dose directly.
• Offline voice. Whisper.cpp (STT, 99 languages) and Piper TTS (~40 languages) run as small local sidecars; if they are not installed the intake screen falls back to typed text and the action plan falls back to browser TTS / on-screen text.

Quick Start

The first launch downloads the Gemma 4 E4B GGUF runtime files and a llama.cpp server binary (~4–5 GB total), npm-installs the frontend, and sets up the offline voice stack. The trained VIGIL LoRA adapter and the LightGBM clinical heads are committed in this repository.

Windows (NVIDIA GPU):

.\run_vigil.ps1

Windows, CPU-only:

.\run_vigil.ps1 -CPU

Skip the offline voice sidecars (faster setup; typed-intake + browser-TTS fallback):

.\run_vigil.ps1 -SkipVoice

Linux, CPU runtime:

bash run_vigil.sh

Linux, Vulkan runtime:

VIGIL_RUNTIME=vulkan bash run_vigil.sh

The app opens at http://localhost:3000. The FastAPI backend runs on :8000, llama.cpp on :8080, and (unless skipped) Whisper.cpp on :8178 and Piper on :5000. Press Ctrl+C in the launcher window to stop everything.

Requirements

• Python 3.10 or newer.
• Node.js 18 or newer.
• Git LFS (the repo uses LFS for the two committed model files — ~42 MB total).
• Windows GPU path: NVIDIA GPU with CUDA 12 support (CPU path works everywhere).
• Linux: CPU runtime works everywhere; Vulkan is selectable when Vulkan drivers are present.

Downloaded at first run

scripts/bootstrap_vigil.py downloads and SHA256-verifies:

File	Source	SHA256
gemma-4-E4B-it-IQ4_XS.gguf	unsloth/gemma-4-E4B-it-GGUF, rev c9ef2c8ea20c9f870ca3e4085aebec77251ff4aa	c932975bbc2f16ac87bae60078d3c7190871a3e8468566166103e0924981f183
mmproj-BF16.gguf	unsloth/gemma-4-E4B-it-GGUF, rev 51a9adf7d1add66b19832d04647cb647381f9294	6d521435bed84c9aade3685f4bc3bce5898dec2b1f1d17f7452ebfaeedc375fb

It also downloads a llama.cpp binary release for the current OS from ggml-org/llama.cpp, and (unless --skip-voice) installs pywhispercpp / piper-tts / imageio-ffmpeg into the venv and pre-fetches a Whisper ggml model plus a handful of Piper voices into runtime/voices/.

Repository contents

run_vigil.ps1 / run_vigil.sh   one-command launcher (llama-server + backend + frontend + voice sidecars)
scripts/
  bootstrap_vigil.py           first-run setup: venv, model + runtime + voice downloads
  serve_whisper.py             minimal offline STT server (whisper.cpp via pywhispercpp)
  serve_piper.py               minimal offline TTS server (Piper)
frontend/                      Vite + TypeScript laptop UI (Material Design 3, 12-lang i18n + RTL, camera overlays)
backend/                       FastAPI clinical API: 3-pass streaming assessment, deterministic tools, encounter persistence
backend/ml/                    deployed LightGBM clinical + jaundice heads (and the calibration prompt)
data/                          WHO reference data (z-score tables, medication doses) and the bundled CHW training curriculum
training/                      dataset generators, LoRA training, GGUF conversion, evaluation scripts
training/vigil-e4b-final/       the trained LoRA adapter (safetensors + config)
training/vigil-e4b-final-language-lora.gguf   the same adapter, llama.cpp-native, applied at runtime
training/eval_reports/         held-out base-vs-VIGIL evaluation reports (JSON + Markdown)
training/TRAINING_GUIDE.md      how to reproduce the LoRA
training/DATASETS.md            the fine-tuning datasets and how to regenerate the corpora

The Android app is released separately in the VIGIL Mobile repository.

Model evaluation

Both evaluators run the deployment path (the IQ4_XS GGUF through llama-server, i.e. exactly what ships):

# baseline: stock Gemma 4 E4B
python training/evaluate_inference.py --run-name base-iq4xs --model-label gemma4-e4b-it-iq4xs

# deployed VIGIL system (LoRA + the two LightGBM heads)
python training/evaluate_gguf_runtime.py --model vigil=gemma-4-E4B-it-IQ4_XS.gguf --lora vigil=training/vigil-e4b-final-language-lora.gguf

evaluate_inference.py needs the backend running; reports are written under training/eval_reports/. The committed reports under training/eval_reports/gguf_runtime/ are the 47-case held-out clinical evaluation referenced in the writeup.

Reproduce the model

See training/TRAINING_GUIDE.md for the full procedure and training/DATASETS.md for the datasets. In short:

python scripts/bootstrap_vigil.py --skip-models --skip-runtime --skip-frontend-install --skip-voice
python -m pip install -r training/requirements.txt
cd training

# 0. build the corpora locally (they are not committed)
python generate_imnci_dataset.py
python prepare_vision_dataset.py            # needs VIGIL_DATASETS_ROOT

# 1. Stage 1 — language / IMNCI LoRA (vision tower frozen) — this is the committed adapter
python train_e4b.py
# 2. Stage 2 — unfreeze the vision tower at a lower LR for jaundice / omphalitis grading
VIGIL_E4B_FINETUNE_VISION=1 VIGIL_E4B_LR=5e-5 VIGIL_E4B_DROP_SOURCES= VIGIL_E4B_NEO_CAP=0 python train_e4b.py

A Colab/Kaggle path is in training/train_on_colab.ipynb.

What is not committed

scripts/bootstrap_vigil.py fetches the base GGUFs, the llama.cpp runtime, the Whisper/Piper binaries and voices, node_modules, and the venv — all .gitignored. The fine-tuning corpora and processed images are regenerated locally (training/DATASETS.md) — the source datasets carry their own licences. Patient encounters and local learner progress under data/ are never committed.

git lfs install      # the .gguf / .safetensors model files use Git LFS
git add .

Clinical tools

Tool	Purpose	Safety
calculate_zscore	WHO weight-for-age z-score (2006 LMS method)	Validated against WHO tables
calculate_medication_dose	Amoxicillin, gentamicin, procaine benzylpenicillin, chlorhexidine	Hard safety caps, never model-generated
calculate_ors_volume	ORS rehydration volume per WHO protocol	3 severity levels
generate_referral_letter	Structured hospital-transfer referral	Includes disclaimer
suggest_followup_date	Next visit per the WHO postnatal schedule	Classification-driven

Safety & ethics

• Decision support only — every output requires review by a qualified health worker or supervisor.
• The system errs toward greater severity on ambiguous findings.
• Hard safety caps on all medication doses; referral is always recommended for WHO danger signs.
• Patient data stays on the device — never transmitted, never committed.

---

Marc Donovici · The Gemma 4 Good Hackathon, 2026