AUDIO- AND LYRICS-BASED MUSIC GENRE CLASSIFICATION USING CNN AND MULTILINGUAL BERT

Abstract

The rapid growth of music content in the digital era has highlighted the need for automated music genre classification systems to replace conventional methods that are less efficient. This study develops a multimodal music genre classification architecture that combines audio signal feature extraction and semantic information from song lyrics in Indonesian and English, focusing on four music genres: Country, Hip-Hop, Pop, and Rock. The proposed deep learning approach integrates a Convolutional Neural Network (CNN) to extract Mel-Spectrogram representations from audio signals and Multilingual BERT to capture contextual relationships within song lyrics. The audio dataset was obtained from the GTZAN Genre Collection, which comprises ten music genres (Blues, Classical, Country, Disco, Hip-Hop, Jazz, Metal, Pop, Reggae, and Rock), while the lyrics dataset was collected from Genius Song Lyrics and Indonesian song lyrics repositories. Features extracted from both modalities were integrated using a feature-level fusion technique. Experimental results demonstrate that the audio-only model achieved an accuracy of 90.4%, while the lyrics-only model obtained an accuracy of 61.7%. The multimodal approach achieved the highest classification performance, with an accuracy of 93.8%. The trained model was subsequently deployed as a web-based application using the Flask framework.