Characterizing actors in online research: Authenticity checking is more than bot detection

<p>Alongside the growing prevalence and flexibility of AI tools, there has been corresponding concern that data collected using online recruitment platforms may increasingly be compromised by “bots.” We tested a variety of countermeasures intended to detect bad-faith behavior on the part of online participants recruited from a Prolific sample (N = 372) and tested using both the Qualtrics and Gorilla research task platforms. Our test suggest that although fully automated bots likely remain rare on Prolific, a substantial minority (21.5%) of participants exhibited patterns of behavior that we characterized as suspicious. Our results suggest that many patterns of behavior that pose a threat to the integrity of data collected online arise not from bots, but from humans who disregard task instructions and make intermittent use of AI and other tools as shortcuts. This renders traditional “attention checks” insufficient to verify good-faith adherence to task instructions. We provide recommendations and next steps for the development of both visible and invisible assays of participant behavior, intended not only to verify their attention to instructions, but also the good-faith authenticity of their responses.</p>