This article discusses the use of explainable AI (XAI) in the field of generative music, where complex machine learning models are often difficult for musicians to understand and control. The paper examines the link between explanation and performance of AI models for the arts, and explores different model architectures, configurations, and training datasets.