Expert validation of AI-generated clinical assessment items for physician assistant certification: a multi-dimensional rubric and dual- reliability method

<title>Abstract</title> <p>Creating high-quality clinical questions for certification exam preparation takes substantial expert time and does not scale to meet learner demand. Generative AI could produce such content at scale, but in high-stakes certification contexts AI-generated content must be checked against the same quality standard as expert-written material, and no established method exists for validating it systematically. We present a validation instrument and method for AI-generated certification exam content, demonstrated on a generation engine we developed for Physician Assistant National Certifying Examination (PANCE) preparation. The approach mirrors the human item-development workflow: items are generated against published item-writing guidelines, then judged by an expert panel against those same guidelines. An initial pilot rounds exposed recurring content problems and guided refinement of the instrument. A second round then validated the refined five-dimension rubric, which use a two-track quality gate separating question construction from explanation quality. In the second round, six PA faculty reviewed 55 items across 14 of the 15 PANCE organ systens. Content validity was strong (overall S-CVI/Ave of 0.9285), and dual inter-rater reliability reporting revealed a prevalence paradox: Fleiss' κ was low at 0.1201 while Gwet's AC1 was high at 0.8653, a gap reflecting skewed rating distribution rather than genuine disagreement. The same paradox appeared in both rounds under different rating designs. In a preliminary blinded student perception study (n = 10), AI and human-written items gave near-identical answering accuracy (70.0% versus 69.3%, p = 0.89) and equivalence clarity and realism, with source detection followed a response-bias pattern rather than real discrimination. The instrument is designed to transfer to any certification exam with a content blueprint, a curated knowledge base, and item-writing guidelines.</p>