Abstract
<title>Abstract</title> <p>Retrieval-Augmented Generation (RAG) architectures combine Large Language Models (LLMs) withexternal knowledge stores to produce grounded, verifiable answers. While RAG systems have beenincreasingly adopted in the biomedical domain, existing work focuses primarily on downstream question-answering (QA) performance, with limited attention to systematic evaluation and usability for non-technical users. This paper presents RAGtio, a modular, open-source framework for the rigorous andreproducible evaluation of RAG retrieval pipelines over domain-specific text corpora, with primary appli-cation to biomedical documents. The framework implements and allows the comparison of four retrievalconfigurations: sparse, dense, hybrid, and hybrid with cross-encoder re-ranking, using a weighted Re-ciprocal Rank Fusion (RRF) mechanism. A key contribution is the introduction of two complementaryevaluation modes: (i) Mode A automatically generates synthetic queries from randomly sampled docu-ment chunks via an LLM, enabling rapid large-scale assessment; (ii) Mode B accepts user-provided queryrelevance pairs assembled from domain expertise, yielding an unbiased estimate of retrieval quality ongenuinely novel queries. The framework is built on the Haystack library and Qdrant vector database,supports heterogeneous document formats, and exposes a YAML-based configuration interface togetherwith a web-based graphical interface for running evaluations and inspecting results without softwaredevelopment expertise.</p>