Evaluating Agreement Between Human Consensus and Large Language Model Ratings of Suicide Reporting

<title>Abstract</title> <p> Responsible reporting on suicide is an important public health strategy but monitoring whether news articles follow reporting recommendations is difficult at scale. While the Tool for Evaluating Media Portrayals of Suicide (TEMPOS) provides a structured approach of rating suicide-related media, human rating requires time, training, and repeated exposure to distressing content. Acknowledging these challenges, our research examines whether large language models (LLMs) can rate incident-focused suicide news articles using TEMPOS with agreement comparable to human raters. We conducted a reliability and agreement study using 42 news articles describing suicide deaths. Seven trained human raters and five LLMs independently rated each article using TEMPOS. Human interrater reliability was assessed using intraclass correlation coefficients (ICCs). A human consensus benchmark, HumanGold, was calculated as the mean TEMPOS score across human raters for each article. Agreement between individual AI models and HumanGold was examined to develop an aggregated AI score, termed AISilver. Individual AI models demonstrated moderate to strong correlations with HumanGold, ranging from <italic>r =</italic>  0.489 to <italic>r =</italic>  0.812. AISilver also showed strong agreement with HumanGold ( <italic>r</italic>  = 0.797), comparable to the highest performing model, Grok ( <italic>r</italic>  = 0.812). Overall, we found that LLMs showed alignment with trained human raters when applying TEMPOS. These findings suggest that AI may be a plausible tool to assist with continuous, large-scale monitoring of suicide-incident reporting, particularly if paired with human oversight. While AI ratings should be interpreted as consensus-aligned screening tools rather than definitive judgments, they may offer a practical path for expanding feedback on suicide-incident reporting. </p>