Attention mechanisms are widely adopted in deep learning for music classification, yet their effectiveness heavily depends on the training strategy of attention mechanisms. In this work, we propose Learnable Counterfactual Attention (LCA), a training strategy that guides attention to focus on truly discriminative, task-relevant regions while suppressing biased or interfering cues in audio signals. LCA explicitly disentangles informative features from spurious correlations during training, leading to consistent performance improvements on singer identification, genre classification, and emotion recognition without additional inference-time cost. Furthermore, we introduce Progressive LCA, a multi-stage refinement scheme that further strengthens attention learning. Our results demonstrate that properly training attention is crucial for both model performance and interpretability in music classification.

Yi-Xing Lin received the Ph.D. degree in computer science and information engineering from National Central University, Taoyuan, Taiwan, in 2025. He is currently a Postdoctoral Research Fellow at the Institute of Information Science (IIS), Academia Sinica, Taiwan. From 2017 to 2020, he worked on attention mechanisms and their applications in natural language processing, including machine translation and question answering. Since 2020, his research has focused on attention-based modeling in audio processing, including non-autoregressive text-to-speech generation and fine-grained music classification. His recent work investigates training strategies for attention mechanisms to enhance interpretability and representation learning in music information retrieval.