Beyond large language models: rediscovering the role of classical statistics in modern data science

Abstract

This study explores the synergy between large language models and classical statistics in contemporary data science. In the field of large language models, we find there is no one-size-fits-all model which satisfies the needs of other scientists. There are differences in the soft results which may be a limitation on their application. To analyze these differences and lack of robustness, we propose a robust methodology that integrates classical statistical experimental design principles with the these advanced models, aiming to identify statistically significant differences among their outcomes. In particular, an experimental design is presented in which the main factors, levels, treatments and interactions that influence the predictions made by different models of complex natural language processing are identified. The main aim of this research is to better understand the influence of some controlled factors that are used in com-plex natural language processing models by applying classical statistical techniques, providing a comprehensive perspective on the relative effectiveness of different zero-shot classification models. It aims to offer practitioners insights into when and where certain models may be more or less sensitive, facilitating informed decision-making in applying these advanced language models. Additionally, computational results obtained from a pilot dataset are presented. These results illustrate the entire process of the proposed methodology, highlighting the importance of considering statistical evidence when making decisions.