Researcher(s)
- Ryan Cortes, Computer Science, University of Delaware
- Shayla Sharmin, Computer Science, University of Delaware
Faculty Mentor(s)
- Roghayeh Leila Barmaki, Computer Science, University of Delaware
Abstract
Large language models are increasingly used as learning tools, and while considerable work has examined their use, there remains scope to understand how the design of AI feedback shapes student learning. This pilot study examines two approaches to AI scaffolding: direct scaffolding, in which feedback explicitly names what a response should include, and indirect (Socratic) scaffolding, in which feedback offers hints and questions that prompt the learner to reflect. In addition, the study examines a second feature of the AI feedback itself—its length. Participants answered introductory questions about human anatomy through a custom web interface connected to a large language model, revising each response across two attempts in response to the AI’s feedback. The study was designed around two feedback features, each varied within subjects. Scaffolding style compared direct against indirect feedback, and feedback length compared brief against extended feedback. To control for order effects, conditions were assigned using a counterbalanced Latin-square design, so that every participant experienced both scaffolding approaches while question and condition order remained balanced across the sample. For each response, answer content, word count, and deliberation time (the interval between receiving feedback and submitting a revision) were recorded, together with eye-tracking data capturing how much attention learners directed toward the feedback. Post-task surveys measured perceived feedback quality and cognitive load (NASA-TLX). By combining behavioral, physiological, and self-report measures, this work compares how direct and indirect AI scaffolding affect the quality and revision of learner responses. Future work will build on these findings to inform how educational AI feedback is designed.