Methodology for analyzing educational forums with NLP: searching for economic terms

Abstract

This chapter studies the programming languages and libraries suitable for presenting a methodology for analyzing forums in the economics subject using natural language processing (NLP) techniques, concluding to use spaCy and transformers in Python. The methodology follows a structure based on CRISP-DM, including project planning and the selection of appropriate tools and technologies. The proposed methodology performs the following actions: Relevant data sources are identified and accessed, collecting data from forum posts, such as text, dates, and authors. Text preprocessing involves noise removal, tokenization, and lemmatization using spaCy, ensuring clean and manageable data. Content analysis begins with calculating the frequency of key terms, followed by topic modeling with techniques like LDA to identify the main discussion topics. Sentiment analysis is performed with transformers models to evaluate the tone of the posts. The results are communicated through visualizations such as word clouds and bar charts, providing a clear understanding of the data. The results are documented in detailed reports that describe the methods used and the interpretations of the findings. Lastly, the results are analyzed and discussed in relation to the initial objectives of the project, offering conclusions and recommendations for future actions or additional studies.