Evaluating Large Language Models for Hardware Vulnerability Discovery: A Controlled Case Study Using SCONE

• Tyler Geist, Computer Engineering, University of Delaware

• Satwik Patnaik, Electrical & Computer Engineering, University of Delaware

Logic locking is a well-established defense against intellectual property piracy and overproduction within the integrated circuit supply chain. Newly proposed logic locking schemes typically undergo rigorous security analysis soon after publication. SCONE, a recent stripped-functionality logic locking scheme, claims resilience against all known input/output-based and structural attacks. However, an independent security analysis subsequently identified a vulnerability in SCONE’s circuit encoding, thereby challenging its claimed resilience. Large language models (LLMs) have recently demonstrated strong performance in hardware security analysis, benchmarking, and the automated generation of attacks and defenses, prompting investigation into their ability to independently identify alternative vulnerabilities. In this study, we used SCONE as a controlled case to evaluate the ability of modern LLMs to rediscover known vulnerabilities. A SCONE executable environment was employed, with Claude Sonnet 5 serving as the assessment agent. Modern LLMs were prompted to analyze the SCONE paper and construct a security review encompassing threat models, concrete attacks, and mitigations. The assessment agent scored LLMs according to a rubric derived from published vulnerabilities, considering both written approaches and any developed code. Gemini 3.6 independently reconstructed the published attack almost completely and achieved the highest score among the models. In contrast, Gemini 3.1 was unable to reproduce the existing attack model but instead proposed a unique, well-specified vulnerability that the assessment agent deemed valid. Because these LLMs showed promise in both reproducing existing security evaluations and discovering unique vulnerabilities, future work should extend this evaluation to additional hardware security benchmarks and logic locking schemes.