To See or to Read: What fMRI Reveals About How Our Brains Process Who Did What to Whom Across Vision and Language

• Stanley Micek, Biological Sciences, University of Delaware

• Alon Hafri, Linguistics and Cognitive Science, University of Delaware

Over the past 30 years, Magnetic Resonance Imaging (MRI) technology has given a non-invasive way to visualize regions and patterns of activity within the brain. This has led to new discoveries about neural computations and the processes that give rise to perception, language, and thought.

Vision and language in particular serve as vital functions in our everyday lives, and recent research has implied that they may be more interconnected than initially thought. At the University of Delaware, this relationship is studied by the Perception and Language (PAL) interdisciplinary lab. One area of interest of the lab is how the brain encodes information determining who did what to whom, such as in the sentence “the girl pushed the boy”, where there is a clear agent (the girl), a verb denoting an action (push), and a patient being acted upon (the boy). But what makes the sentence “dog bites man” mundane, but the sentence “man bites dog” unusual? And how are we able to form and comprehend the difference?

During the 2026 INBRE Summer Scholar Program, I have had the opportunity to work with the PAL lab on an ongoing study regarding the patterns of neural activity that result from visual or written stimuli in interactions involving who did what to whom. This experience has allowed me to learn about fMRI studies as both a researcher (e.g. MRI safety, surrounding literature, etc.) and a participant (e.g. being the study pilot). 

In this study, participants complete tasks related to agent and patient stimuli while being scanned inside of the MRI. This agent and patient stimuli would either be visual stimuli (e.g. a 3D rendered image of a girl pushing a boy) or written stimuli (e.g. the sentence “the girl pushed the boy” on a grey background). This data will then be processed through multi-voxel pattern analysis (MVPA) to determine if regions that encode who did what to whom in language are spatially distinct from those that encode information from visual scenes.