If it bleeds, it leads? Evaluating whether LLMs can identify what is newsworthy

<p>Journalists have always had to make the challenging decision about which stories to focus on and which stories to ignore. This paper investigates whether AI can assist journalists in identifying newsworthy stories from large databases. We first build a dataset of Canadian lobbying registrations, each labeled as "newsworthy" or "not newsworthy" based on whether a journalist covered it in a published article or not. We then systematically evaluate several AI models' ability to identify the one newsworthy registration within a batch of ten. The best models identify a single newsworthy registration with 22% accuracy, rising to 48% when allowed to select three, and to 52% when provided previous headline examples. Our results suggest that a filtering approach silently discards real stories, whereas allowing the model to rank registrations by newsworthiness ensures no information is lost. We conclude that while AI holds real potential to help identify leads, this is not possible without further investment in evaluation infrastructure, and even then, there is an important role for journalists to identify what AI misses, as it cannot replace their editorial judgment entirely.</p>