Goodness-of-fit tests for categorical models of psychological processes: Fixing the occasional failures of asymptotic theory

Abstract

The goodness of fit of categorical models of psychological processes is often assessed with the log-likelihood ratio statistic (G2), but its underlying asymptotic theory is known to have limited empirical validity. We use examples from the scenario of fitting psychometric functions to psychophysical discrimination data to show that two factors are responsible for occasional discrepancies between actual and asymptotic distributions of G2. One of them is the eventuality of very small expected counts, by which the number of degrees of freedom should be computed as (J 1) × I P K0.06, where J is the number of response categories in the task, I is the number of comparison levels, P is the number of free parameters in the fitted model, and K0.06 is the number of cells in the implied I × J table in which expected counts do not exceed 0.06. The second factor is the administration of small numbers ni of trials at each comparison level xi (1 ≤ i ≤ I). These numbers should not be ridiculously small (i.e., lower than 10) but they need not be identical across comparison levels. In practice, when ni varies across levels, it suffices that the overall number N of trials exceeds 40 × I if J = 2 or 50 × I if J = 3, with no ni lower than 10. Correcting the degrees of freedom and using large ni are easy to implement in practice. These precautions ensure the validity of goodness-of-fit tests based on G2.