When evaluating my system for faults or opportunities for improvement, I listen critically, typically to well-culled tunes with ranges of frequencies. When exploring new music I want something that touches me in an emotional way. And if it strikes me in a good way it immediately goes into a playlist for further culling in or out later. And then sometimes a nice glass of bourbon and hitting random play is equally enjoyable (Calgon, take me away). Those are my 3 categories of listening depending on my mood.
"audiophiles listen to their equipment"
That quote is misattributed to Alan Parsons, as I understand. Anyway saying there's a problem with wanting good sound is like saying an instrument designer, aiming for beautiful sound, could not possibly be a music lover and is only interested in the sound of the instruments. I.e. the sound is inseparable from the music. For me the beauty of the sound, good microdynamics, and so on, are the "doors" to the meaning of the music.