Linking image saliency with modulations along the visual cortex

Analysis of the Natural Scenes Dataset (NSD) to understand the relation between static image saliency and retinotopic activity in the visual, parietal, and prefrontal region in the brain. The goal is to high vs low‐level model predictions for spatial modulations in the cortex.

NSD is a large-scale fMRI dataset containing fMRI measurements of 8 healthy adult subjects viewing of color natural scenes from the COCO dataset over the course of 30–40 scan sessions.

Here, we have used a generalized model for predicting image saliency, such as DeepGaze, to generate 2D saliency maps for each of the stimuli shown to the subjects.

The notebooks in the project are structured according to the general flow of the exploration and analysis:

• 00_Exploration_DeepGaze_COCO - Exploring the NSD dataset stimuli and generating 2D saliency maps with DeepGaze
• 01_pRF_single_subj - Exploring the population receptive field (pRF) model and visualize sanity checks for a single subject
• 02_pRF_sanity_check_multisubj - Sanity checks for multiple subjects for visualizing the trend between pRF sigma and eccentricity
• 03_pRF_saliency_modelling - Modelling how each pixel in the NSD stimuli and saliency map, maps to the pRF parameters (visual field representation in the brain)

Get this project

$ git clone https://github.com/hrdkbhatnagar/nsd-static-saliency.git

Environment

This project enviroment was created using Conda, within a cloud computing cluster. The entire environment required for this project is described in the environment.yml located in the main directory. The enviroment can be recreated using:

$ conda env create --name NSD-Saliency --file environment.yml

• Python version 3.8.13 was used for writing the code under JupyterLab version 3.4.8

Structure

• data/: Contains the generated intermediate datafiles for the project. (Note: Currently not included due to GitHub size restrictions. However they can be generated by running the notebooks in order)

• src/: Contains reusable Python modules for the project, including modified DeepGaze code.

• results/: Contains figures and saved model files.

• scripts/: Contains all the Jupyer notebooks and Python scripts.

Related studies

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., Hutchinson, J. B., Naselaris, T., & Kay, K. (2022). Nature Neuroscience, 25(1), Article 1

State-of-the-Art in Human Scanpath Prediction Kümmerer, M., & Bethge, M. (2021) (arXiv:2102.12239)

Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex .St-Yves, G., Allen, E. J., Wu, Y., Kay, K., & Naselaris, T. (2022). Neuroscience