I think that staying focused only on the music is hard for many of the reasons that folks have shared, But I also think that it is something we can gently push back on if we would like to stay more music focused. I was lousy at meditation - my mind drifted towards my random thoughts so easily, and I was frustrated by what a battle it was to keep focusing on my breathing.
But focusing on music is actually easier for me. When I do inevitably drift away towards some thoughts or worries, gently bringing myself back to the music is easier and more fulfilling than bringing myself back to my breathing. Also, for the last few years, I write something about each piece of music I listen to in my listening room. I keep a journal next to the chair, and I find that listening, and then writing about what I just heard (one album side at a time, usually) keeps me in conversation with myself about music. For me, it helps.
If that all seems like too much work, I understand. There is certainly no harm done doing more than one thing, if you like them both. We really only do one thing at a time, anyway. We just switch focus so rapidly that it seems like more than one...