Well I think this is part of the problem with double blind and similar listening tests. Listening for a difference is the worst type of critical listening. Further, you know you are being tested so it changes the entire experience.
"audiophiles listen to their equipment"
That quote is misattributed to Alan Parsons, as I understand. Anyway saying there's a problem with wanting good sound is like saying an instrument designer, aiming for beautiful sound, could not possibly be a music lover and is only interested in the sound of the instruments. I.e. the sound is inseparable from the music. For me the beauty of the sound, good microdynamics, and so on, are the "doors" to the meaning of the music.