Cross-Dataset Analysis of Language Models for Generalised Multi-Label Review Note Distribution in Animated Productions

Abstract

During the production of an animated film, supervisors and directors hold daily meetings to evaluate in-progress material. Over the course of the several years it takes to complete a film, thousands of text notes outlining required fixes are generated. These notes are manually allocated to various departments for resolution. However, as with any manual process, a significant number of notes are either delayed, miss-assigned or overlooked entirely, which can negatively impact the final quality of the film. This paper investigates the performance of various methods for automating the distribution of review notes across relevant departments using datasets from multiple films produced by an animation studio in Madrid, Spain. Since each note can belong to multiple departments, the task is posed as a multi-label classification problem. The analysis and comparison of the results obtained with datasets from three different films, focusing on generalisation, provides critical insights for any Animation Studio evaluating the use of these methods in their process. The methods leverage Large Language Models (LLMs), including encoder-only models such as BERT and decoder-only models like Llama 2. Fine-tuning with QLoRA and in-context learning techniques were applied and evaluated across all datasets, and a cross-dataset analysis is presented. The fine-tuned encoder-only model achieved an F1-score of 0.98 for notes directed to the Animation department. Training was carried out locally on an RTX-3090 GPU, completing it in less than 30 min.