Can Large Artificial Intelligence-Based Linguistic Models Help to Obtain Information About Burning Mouth Syndrome?

Abstract

Objective: Burning Mouth Syndrome (BMS) is an idiopathic chronic orofacial pain disorder with diagnostic and therapeutic challenges. Inexperienced clinicians may desperately resort to online information. The objective of this study was to evaluate the usefulness, quality, and readability of responses generated by three artificial intelligence large language models (AI-LLMs)-ChatGPT-4, Gemini, and Microsoft Copilot-to frequent questions about BMS.

Materials and methods: Nine clinically relevant open-ended questions were identified through search-trend analysis and expert review. Standardized prompts were submitted, and responses were independently rated by 12 international experts using a 4-point usefulness scale. Quality was evaluated using the QAMAI tool. Readability was measured using Flesch-Kincaid Grade Level and Reading Ease scores. Statistical analyses included Kruskal-Wallis and Bonferroni correction.

Results: All AI-LLMs produced moderately useful responses, with no significant difference in global performance. Gemini achieved highest overall quality scores, particularly in relevance, completeness, and source provision. Copilot scored lower in usefulness and source provision. No significant differences were obtained among AI-LLMs. Average readability corresponded to 12th grade, with ChatGPT requiring the highest proficiency.

Conclusions: AI-LLMs show potential for generating reliable information on BMS, though variability in quality, readability, and source citation remains concerning. Continuous optimization is essential to ensure their clinical integration.