Codebook Scaffolding Drives Reliable Coding of Chemistry Misconceptions by Large Language Models: A Controlled, Multilingual, Multi-Topic Benchmark

<jats:p>Diagnosing students’ chemistry understanding requires judging whether a statement expresses a specific misconception, a correct idea, or an ambiguous claim. Such classification is central to feedback, teaching, and chemistry education research, but depends on expert interpretation of meaning and boundaries. This study evaluated whether large language models (LLMs) can support such coding under controlled conditions. We tested 280 literature-informed, single-idea claims on acids and bases and the particulate nature of matter. Claims were analysed in English, Croatian, and Slovenian using three LLMs, two prompt conditions, and five independent stateless runs, producing 25200 classifications. Pre-service teacher (PST) coding provided a human reference. Prompt structure was decisive. Majority-vote agreement with expert-adjudicated gold codes was only 16.94% when LLMs received code labels alone. Adding explicit codebook definitions, operational rules, and boundary criteria increased agreement to 91.79%, and 95.00% of scaffolded cells were unanimous across five runs. However, stable outputs were not always correct: 55.1% of scaffolded errors repeated across all five runs and all three models. Remaining LLM errors concentrated at ambiguity boundaries: correct claims were never miscoded as misconceptions, whereas false-positive misconception coding came from ambiguous claims. PST coding matched gold in 70.8% of individual classifications and 84.3% after collapsing codes into misconception, correct, or ambiguous groups. These findings support LLMs as coding assistants in controlled chemistry misconception coding, provided outputs are anchored in explicit codebooks and reviewed through chemistry education judgement.</jats:p>