Internet search "room colors and emotions" and you will find an endless number of articles on the subject. There's no agreement as to the exact mechanism, but it's a causal link that's been exploited by advertisers, interior designers, institutions (hospitals, schools, etc.) and film makers for decades. Since anything that can effect your moods and emotions can effect the experience of listening to music, then yes, room colors matter.