Beyond the null: Calibration, balance, and the interpretation of Pillai scores

<p>Pillai scores have been a standard tool in linguistics to measure vowel overlap for more than two decades, but only recently has the metric's behavior come under scrutiny. Current standards of practice rest on a null condition: where the two underlying distributions overlap completely. However, these standards do not hold for most naturalistic data, where the underlying distributions are unknown. For imbalanced data, the ratio of tokens in the two categories does two jobs at once: it governs researchers' ability to detect a real distinction as a function of the harmonic mean of the class sizes, and it governs the value that the Pillai statistic converges on. Speakers with differently balanced samples therefore estimate different targets — as much as 4.68-fold at a 5:95 ratio and a separation of 0.8 within-class standard deviations. Downsampling to equal totals is often employed as a control but does not fix the problem, since doing so equalizes the wrong quantity. A Pillai score below a given threshold is thus not evidence of merger. Instead, a minimum detectable separation and an equivalence test against a stated bound, both from the same noncentral Beta distribution, should be reported alongside it.</p>