Researcher(s)
- Asher Mills, Computer Science, University of Delaware
Faculty Mentor(s)
- Kathleen McCoy, Computer and Information Sciences, University of Delaware
Abstract
Optical Character Recognition (OCR) has experienced significant advancements with the emergence of large vision-language models (LVLMs), which combine visual understanding with natural language generation. Traditional OCR systems were designed specifically to extract text from images, whereas modern multimodal models are capable of performing OCR as part of broader visual reasoning tasks. This study evaluates the effectiveness of both approaches on text recognition in images captured by Blind and Low-Vision (BLV) individuals using the VizWiz Visual Question Answering dataset. We compare proprietary and open-source LVLMs, including GPT-5 Mini and Qwen2-VL, against traditional OCR systems, Apple Vision OCR and Tesseract on a subset of this dataset focusing on text recognition.
To improve the open-source model’s OCR capabilities, Qwen2-VL (2B parameters) was fine-tuned using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA), allowing only a small subset of model parameters to be updated while substantially reducing computational and memory requirements. This fine-tuned model was also compared against the base model of Qwen2-VL. Fine-tuning and inference evaluation were performed on the University of Delaware High Performance Computing (HPC) cluster, while GPT-5 Mini was evaluated through the OpenAI API. Model performance was measured using the VizWiz accuracy metric, with containment-based matching to account for the generative nature of LVLM responses.
Our findings demonstrate a significant performance advantage of modern LVLMs over conventional OCR pipelines and highlight the potential of lightweight, fine-tuned open-source models to rival or exceed proprietary alternatives in real-world accessibility applications involving BLV imagery.