How Large Language Model Benchmarks Are Built: A Measurement-Oriented Tutorial and Empirical Demonstration with IRT-Viz-Eval

<p>Benchmark construction differs from developing an educational or psychologicalassessment. This tutorial introduces LLM benchmark development to social-sciencereaders, especially those working in psychological and educational measurement, througheight stages from intended use to maintenance. IRT-Viz-Eval illustrates the processwith 84 generated psychometric plots and 198 tasks. Five vision-language systems wereadministered through Hugging Face Inference Providers and scored against program-matic ground truth. Mean task-normalized scores ranged from .600 to .719; task profiles,output validity, latency, and aggregation differed. The demonstration shows why in-terpretation requires more than a leaderboard total. Validity and scale-developmentscholarship disciplines benchmark claims, but benchmark scores are not treated aspsychological traits, and the paper does not assume that every benchmark requires ahuman-test psychometric model.</p>