Researcher(s)
- Aru Dubey, Neuroscience, University of Delaware
Faculty Mentor(s)
- Timothy Vickery, Psychological and Brain Sciences, University of Delaware
Abstract
When we try to recognize what we are looking at, the visual system breaks a
scene into separate objects and surfaces. It is still unclear the extent to
which the visual brain explicitly tracks this parsing, or whether it can be fully
explained by two things we already know the visual system encodes: what the
objects are, and where the edges fall. We measured segmentation structure in
images using Segment Anything Model (SAM), a model that divides an image into
regions. From its output we took simple image-level measures: how many regions
an image contains, how much boundary they create, how confident the model
was, and how the regions are arranged. We then asked whether these measures
predict brain activity that other models cannot. We used the Natural Scenes
Dataset, in which eEight participants each viewed about 10,000 natural scenes
during 7T fMRI. In V1, V2, V3, and hV4, we fit models
predicting each voxel’s response and compared three feature sets: SAM
segmentation, CLIP (object meaning), and Gabor filters (edges and contrast).
Segmentation explained variance the other two could not, in every region and in
all 8 participants. Whole-image measures, however, are the same for every voxel,
so they cannot show where in the visual field the effect arises. We therefore
recomputed segmentation within each voxel’s receptive field and compared it
against an edge model measured in that same location. Segmentation
explained unique variance, in all 8 participants and in 96–98% of voxels, and the
effect was about four times larger than the whole-image version. Our results
suggest that higher-order aspects of segmentation are encoded in early-to-middle
visual cortex, over and above what can be explained by coding of edges and
contrast. Future work will examine effects in a broader set of visual regions,
additional datasets, and according to specific types of segmentation statistics.