MindPilot

Closed-loop Visual Stimulation Optimization for Brain Modulation with EEG-guided Diffusion

---

📰 News

• 🚀 2026.03.08: We update the codebase and Hugging Face resources.
• 📄 2026.02.11: We upload the arXiv paper and release the code repository.
• 🎉 2026.01.26: Our paper is accepted to ICLR 2026.

📖 Overview

We propose MindPilot, which employs a simple black-box optimization approach to achieve EEG-guided closed-loop visual stimulation optimization (supporting offline learning data and experience replay) for regulating the brain activity of subjects. This paper addresses three problems: high-noise non-invasive brain signals; the non-differentiability of brain input and output; and the variability of brain activity states.

🎯 Conceptualization

The conceptualization of MindPilot: Closed-loop optimization of visual stimuli using EEG feedback

---

🏗️ Architecture

Overall architecture of MindPilot framework

Installation

Option 1: Quick Setup (Recommended)

cd MindPilot
chmod +x setup.sh
./setup.sh
conda activate MindPilot

Option 2: Manual Setup

conda env create -f environment.yml
conda activate MindPilot

---

📦 Pretrained Weights & Datasets

We provide pretrained model weights and preprocessed datasets on Hugging Face:

🤗 https://huggingface.co/datasets/LidongYang/Mindpilot

You can download using the Hugging Face CLI:

# Install huggingface_hub if not already installed
pip install huggingface_hub

# Download all files
huggingface-cli download LidongYang/Mindpilot --repo-type dataset --local-dir ./data

External Datasets

Download the additional datasets from the following sources:

Dataset	Description	Download Link
THINGS-EEG2	Natural images with EEG responses	OSF
ArtPhoto	Artistic photographs with emotion ratings	ImageEmotion
GAPED	Geneva Affective Picture Database	UNIGE
EmoSet	Large-scale emotion dataset	VCC Tech

---

🎓 Usage

1. Train EEG Readout Model

Train a neural network to predict EEG responses from visual features:

python model/end_to_end.py \
    --dnn alexnet \
    --sub 10 \
    --modeled_time_points all \
    --pretrained False \
    --epochs 50 \
    --lr 1e-5 \
    --weight_decay 0. \
    --batch_size 64 \
    --save_trained_models True \
    --project_dir eeg_encoding/

2. Interactive Search

Perform evolutionary search in the latent space for optimal stimuli:

jupyter notebook experiments/exp-interactive_search.ipynb

3. Heuristic Generation

Generate optimized visual stimuli using gradient-based optimization:

python experiments/exp-heuristic_generation_with_guidance_anyfeature.py

4. Benchmark Evaluation

Offline Generation Benchmark

bash experiments/run_benchmark_offline_generation.sh

Heuristic Generation Benchmark

bash experiments/run_benchmark_heuristic_generation.sh

Complete Benchmark Suite

bash experiments/run_benchmark_total.sh

5. Real-time Human Experiments

Server side (runs optimization):

python server/improved_experiment.py --port 5000

Client side (presents stimuli and records EEG):

python client/client.py --server_ip 192.168.1.100 --port 5000

📝 Citation

If you find this work useful, please cite our paper:

@article{li2024visual,
  title={Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion},
  author={Li, Dongyang and Wei, Chen and Li, Shiying and Zou, Jiachen and Liu, Quanying},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={102822--102864},
  year={2024}
}
@inproceedings{2026mindpilot,
  title={MindPilot: Closed-loop Visual Stimulation Optimization for Brain Modulation with {EEG}-guided Diffusion},
  author={Dongyang Li, Kunpeng Xie, Mingyang Wu, Yiwei Kong, Jiahua Tang, Haoyang Qin, Chen Wei, Quanying Liu },
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=7jdmXx869Q}
}

😺 Acknowledgement

We sincerely thank the following works and their authors for providing valuable resources that supported this project:

• The THINGS-EEG2 dataset — "A large and rich EEG dataset for modeling human visual object recognition" by Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, and Radoslaw M. Cichy.
• FAST-DiReCT — "Query-Efficient Online Black-box Guidance for Diffusion-Model Target Generation" by Kim Yong Tan, Yueming Lyu, Ivor Tsang, and Yew-Soon Ong.
• D3PO — "Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model" by Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li.
• DDPO — "Training Diffusion Models with Reinforcement Learning" by Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine.
• DPOK — "Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models" by Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, Kimin Lee.

📬 Contact Dongyang Li if you have any questions or suggestions.

📜 License

This project is licensed under the MIT License.