OcuFusion: Multimodal Retinal Imaging Biomarker Explorer

Overview

OcuFusion is a Python, PyTorch, and Streamlit research prototype for exploring multimodal ocular imaging fusion. It combines fundus-style retinal images, OCT-style B-scans, and tabular variables into an interpretable exploratory biomarker-style risk profile.

The project is self-contained: it ships with a synthetic/demo dataset generator, a multimodal neural model, missing-modality handling, explainability utilities, modality ablation, evaluation scripts, and an exportable reporting workflow.

Why this project matters

Real-world ocular imaging workflows often involve multiple data streams. A case may have fundus photography, OCT imaging, and structured clinical variables, but one or more modalities may be missing, lower quality, or unavailable. OcuFusion demonstrates how a research pipeline can:

• fuse image and tabular modalities;
• remain robust when fundus or OCT inputs are missing;
• provide transparent model-behavior visualizations;
• compare predictions across modality combinations;
• package results into exportable research-style reports.

This makes the repository a compact example of responsible multimodal prototype design.

Key features

• Multimodal fusion of fundus-style images, OCT-style images, and tabular variables.
• Built-in synthetic dataset, so no private medical data is required.
• PyTorch dataset class with modality masks and random modality dropout.
• CNN encoders for fundus and OCT, plus an MLP encoder for tabular inputs.
• Missing-modality support for fundus-only, OCT-only, tabular-only, and mixed inputs.
• Fundus Grad-CAM and OCT saliency visualizations.
• Tabular perturbation feature importance.
• Modality ablation across seven modality combinations.
• Streamlit dashboard with upload/sample case support.
• Training, evaluation, confidence summaries, robustness metrics, and report export.
• Model card documenting intended use, synthetic data assumptions, and limitations.

Demo dataset

The demo dataset is generated synthetically with NumPy, Pillow/OpenCV-style drawing, and controlled hidden scoring logic.

Each case includes:

• fundus_path: synthetic RGB fundus-style image.
• oct_path: synthetic grayscale OCT-style B-scan.
• tabular fields: age, sex, visual acuity, retinal thickness, vessel-density estimate, image quality score, and signal strength.
• synthetic_biomarker_score: normalized synthetic target score in [0, 1].
• risk_class: low, moderate, or high class derived from the synthetic score.
• ood_flag: synthetic out-of-distribution marker.

Generated files live under:

data/demo/
  fundus/
  oct/
  metadata.csv

Optional public ophthalmology datasets can be placed manually under data/external/, but the project never depends on external data to run.

Model architecture

OcuFusion uses three branches:

• Fundus CNN encoder: small convolutional encoder producing a 128-dimensional embedding.
• OCT CNN encoder: OCT-adapted convolutional encoder producing a 128-dimensional embedding.
• Tabular MLP encoder: normalized tabular features mapped to a 64-dimensional embedding.

The fusion model concatenates:

fundus_embedding + oct_embedding + tabular_embedding + modality_mask

The fused representation feeds:

• a regression head for the exploratory ocular biomarker score;
• a class-probability head for low/moderate/high output;
• a confidence estimate from predicted class probability;
• a modality contribution gate for fundus, OCT, and tabular branches.

See MODEL_CARD.md for intended use, data assumptions, evaluation notes, and responsible-use boundaries.

Missing-modality support

The dataset and model both support missing modalities. The modality_mask has three entries:

[fundus_available, oct_available, tabular_available]

When a modality is unavailable, its embedding is replaced with zeros and the mask informs the fusion network which branches are present. During training, optional modality dropout simulates missing fundus or OCT inputs:

training:
  modality_dropout: true
  drop_fundus_prob: 0.15
  drop_oct_prob: 0.15

The model supports:

• all modalities;
• fundus + tabular;
• OCT + tabular;
• fundus only;
• OCT only;
• tabular only;
• fundus + OCT.

Explainability

OcuFusion includes model-behavior explanation utilities:

• fundus_gradcam(...): fundus image, Grad-CAM heatmap, and overlay.
• oct_saliency_map(...): OCT image, saliency heatmap, and overlay.
• perturb_tabular_feature_importance(...): tabular feature ranking by score change.
• quality checks for sharpness, brightness, contrast, missing inputs, and unusual tabular ranges.

These methods are intended to help inspect model behavior in a research prototype. They are not evidence of real-world ocular findings.

Modality ablation

A standout feature is case-level modality ablation. The same case can be evaluated under:

• all modalities;
• fundus only;
• OCT only;
• tabular only;
• fundus + OCT;
• fundus + tabular;
• OCT + tabular.

The dashboard displays score and confidence shifts across these combinations so reviewers can see whether multimodal fusion changes the model output or improves prediction stability.

Streamlit dashboard

Run the app with:

streamlit run app.py

The dashboard includes:

1. Upload / Sample Case
2. Multimodal Prediction
3. Fundus Explainability
4. OCT Explainability
5. Tabular Feature Contribution
6. Dataset Quality Summary
7. Model Evaluation
8. Exportable Case Report

It supports built-in sample cases, custom fundus/OCT uploads, manual tabular feature entry, Plotly charts, visual explanation panels, modality ablation, and downloadable Markdown/HTML/JSON reports.

Installation

git clone <repo-url>
cd ocufusion
python -m venv .venv
source .venv/bin/activate  # Mac/Linux
# or .venv\Scripts\activate on Windows

pip install -r requirements.txt

Quick start

python -m src.data.generate_demo_dataset
python -m src.models.train --epochs 10 --batch-size 32
python -m src.models.evaluate
streamlit run app.py

For a faster smoke test:

python -m src.models.train --epochs 1 --batch-size 32 --max-cases 64
python -m src.models.evaluate
streamlit run app.py

Training

Training loads the demo dataset, creates train/validation/test splits, applies random modality dropout when configured, and optimizes a combined loss:

loss = regression_loss + classification_loss

Outputs are written to:

outputs/models/
  ocufusion_demo_model.pt
  training_history.csv
  training_config.yaml
  splits/

Default command:

python -m src.models.train --epochs 10 --batch-size 32

Evaluation

Evaluation computes:

• regression metrics: MAE, RMSE, R2;
• classification metrics: accuracy, balanced accuracy, macro F1, confusion matrix;
• confidence metrics: average confidence, low-confidence count, confidence/error summary;
• modality robustness metrics across all modality combinations.

Run:

python -m src.models.evaluate

Outputs are written to:

outputs/models/evaluation_metrics.json
outputs/models/evaluation_predictions.csv
outputs/figures/demo_case_prediction.png
outputs/figures/modality_ablation_chart.png
outputs/figures/predicted_vs_true_biomarker_score.png
outputs/figures/predicted_vs_true_score.png
outputs/figures/confusion_matrix.png
outputs/figures/modality_ablation_performance.png
outputs/figures/dataset_summary.png
outputs/figures/confidence_vs_error.png

Exportable reports

The dashboard can export case reports in:

• Markdown;
• HTML;
• JSON.

Reports include:

• case ID;
• available and missing modalities;
• exploratory biomarker score;
• low/moderate/high probabilities;
• model confidence;
• image-quality warnings;
• modality contributions;
• top tabular features;
• modality ablation summary;
• responsible-use disclaimer.

Reports are saved to:

outputs/reports/

Project structure

ocufusion/
  app.py
  README.md
  MODEL_CARD.md
  requirements.txt
  config/
    default.yaml
  data/
    demo/
      fundus/
      oct/
      metadata.csv
    external/
      README.md
  notebooks/
    01_demo_dataset_preview.ipynb
    02_training_experiment.ipynb
  src/
    data/
      generate_demo_dataset.py
      dataset.py
      preprocessing.py
      quality_checks.py
    models/
      encoders.py
      fusion_model.py
      train.py
      evaluate.py
      predict.py
    explainability/
      gradcam.py
      saliency.py
      feature_importance.py
      modality_ablation.py
    reporting/
      report_generator.py
    utils/
  outputs/
    models/
    reports/
    figures/
  tests/

Example outputs

After running training and evaluation, reviewers can inspect:

• outputs/models/training_history.csv for epoch-level training curves;
• outputs/models/evaluation_metrics.json for model metrics and modality robustness;
• outputs/figures/demo_case_prediction.png for a static example case profile;
• outputs/figures/confusion_matrix.png for class behavior;
• outputs/figures/predicted_vs_true_score.png for regression behavior;
• outputs/figures/modality_ablation_chart.png for robustness comparison;
• outputs/figures/dataset_summary.png for demo evaluation summary visuals;
• exported case reports under outputs/reports/.

The Streamlit dashboard surfaces these outputs interactively for screenshots and exploratory review. Manual dashboard screenshots can be added later under outputs/figures/screenshots/ or a future docs/screenshots/ folder.

Static Evaluation Figures

Limitations and disclaimer

OcuFusion is a software research prototype, not a medical device, and is not intended for care planning, treatment recommendations, or real-world clinical interpretation. The default dataset is synthetic and intended for software demonstration only.

The demo score, risk classes, and quality warnings are generated for engineering and research-prototype illustration. They should not be interpreted as medical evidence or used for patient care.

Any real research deployment would require appropriate data governance, ethics review, dataset documentation, external validation, bias analysis, uncertainty analysis, clinical collaboration, and regulatory review where applicable.

Future work

• Add support for real public benchmark datasets with documented loaders.
• Add calibration plots and uncertainty estimation beyond softmax confidence.
• Add cross-validation and experiment tracking.
• Add richer OCT layer simulation and image-quality augmentation.
• Add optional pretrained image backbones.
• Add model cards and dataset cards.
• Add continuous integration for tests and linting.
• Add containerized deployment for reproducible demos.

Tests

Run:

pytest

The test suite covers dataset generation/loading, missing modalities, model forward passes, modality ablation, explainability utilities, and Markdown/HTML/JSON report generation.