Where Does Negation Understanding Break Down in Generative Vision-Language Models? A Vision-Encoder Probing Study

• Khadija Mohammadi, Computer Science, University of Delaware

• Rahmat Beheshti, Computer & Information Science, University of Delaware

Where Does Negation Understanding Break Down in Generative Vision-Language Models? A Vision-Encoder Probing Study

Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet they continue to struggle with linguistic negation. For example, a model may correctly recognize “there is a dog” but fail to distinguish it from “there is no dog.” Such errors are especially concerning in medical imaging, where confusing the presence and absence of clinical findings can lead to incorrect diagnoses. While most prior work on negation has focused on contrastive VLMs and general-domain datasets, generative medical VLMs remain largely unexplored. More importantly, it is still unclear where negation understanding breaks down within these models. Before developing effective solutions, we must first determine whether the vision encoder fails to represent the presence or absence of clinical findings, or whether this information is already encoded in the visual embeddings and is instead lost during language-side processing. To answer this question, we isolate the vision encoder of LLaVA-Med and evaluate, independently of the language model, whether its frozen image embeddings linearly encode the presence versus absence of clinical findings. Using balanced positive and negative chest X-ray images for thirteen CheXpert conditions from MIMIC-CXR-JPG, we train a separate linear probe for each condition (e.g., Edema) to classify the corresponding image embeddings as representing either the presence or absence of that finding. Strong probe performance would indicate that the vision encoder already preserves the necessary clinical information, suggesting that negation failures primarily arise during language-side reasoning rather than visual perception. Our findings establish a simple diagnostic framework for localizing negation failures within generative medical VLMs and provide a foundation for developing more targeted and effective methods to improve negation understanding.