Generating News with Large Language Models: A Human-AI Pipeline for Controlled Media Experiments

• Wei-en Tan, Computer Science, University of Delaware

• Matthew Mauriello, Computer and Information Science, University of Delaware

News media plays a critical role in shaping public understanding of complex scientific issues such as climate change. Prior research has shown that differences in how scientists are portrayed can substantially influence readers’ trust, perceived credibility, and support for science-informed policies. However, studying these effects experimentally presents a longstanding challenge. Researchers must create multiple versions of the same news article that differ only along carefully controlled dimensions while preserving factual content, structure, readability, tone, and writing quality. Traditionally, these parallel stimuli are created through extensive manual rewriting, a process that is time-consuming and difficult to scale.

This project investigates whether large language models (LLMs) can serve as research instruments for generating high-fidelity news articles suitable for controlled behavioral experiments. We leverage LLMs within a human-in-the-loop workflow to produce parallel versions of climate news articles that preserve the original facts while manipulating only two theoretically motivated dimensions: (1) how scientists are portrayed (neutral, engaged, or advocacy-oriented figures) and (2) whether scientific evidence emphasizes consensus or uncertainty. Human review and quality assurance ensure that the generated articles maintain factual fidelity and experimental control.

Using these validated stimuli, we will conduct a between-subjects experiment in which participants read one version of a climate news article and complete measures of trust in science, perceived credibility of scientists, perceived scientific consensus, and support for climate-related policies. We anticipate that portraying scientists as engaged experts, communicating concern while remaining evidence-based will increase trust and credibility relative to strongly advocacy-oriented portrayals.

Beyond climate communication, this work contributes a reusable human-AI framework for generating controlled experimental stimuli. By demonstrating how LLMs can augment researchers in producing consistent, scalable, and theoretically grounded counterfactual media content, this lays the foundation for future experimental research on misinformation, political communication, health communication, and other domains where precise manipulation of communication is essential.