Abstract
<title>Abstract</title> <p>Understanding the intention of an utterance is crucial in conversational communication. As conversational artificial intelligence models are rapidly being developed and applied in various fields, it is important to evaluate the LLMs’ capability to understand the intentions of user’s utterance. Speech act is a linguistic concept in pragmatics, the study of human language use, and can be simply understood as referring to the intention of an utterance. This study evaluates whether current LLMs can understand the intention of an utterance by considering the given conversational context, particularly in cases where the actual intention differs from the literal intention of the sentence, i.e. indirect speech acts. With a specific focus on Korean, a context-sensitive language, we construct evaluation datasets, consisting of three types of indirect speech acts based on Searle’s speech act scheme. The dataset consists of two scenarios, where the same utterance conveys direct speech act and indirect speech act respectively, depending on the conversational context. We adopt two experimental setups: the conventional Multiple-Choice Question (MCQ) format for automatic evaluation, and the Open-Ended Question (OEQ) for detailed assessment by human experts. For thorough evaluation, we also conduct MCQ-based experiment with human participants to set gold standard. Our findings reveal that Claude3-Opus outperformed the other competing models, with 71.94% in MCQ and 65% in OEQ, showing a clear advantage. In general, proprietary models exhibited relatively higher performance compared to open-source models. Nevertheless, no LLMs reached the level of human performance. Most LLMs, except for Claude3-Opus, demonstrated significantly lower performance in understanding indirect speech acts compared to direct speech acts, where the intention is explicitly revealed through the utterance. This study not only performs an overall pragmatic evaluation of each LLM’s language use through the analysis of OEQ response patterns, but also emphasizes the necessity for further research to improve LLMs’ understanding of indirect speech acts for more natural communication with humans.</p>