No. It's about the music, not the recording. If it wasn't about the music most people would never go to a concert unless they were able to find the best seat for sound quality and were able to get it each time.
Does it have to sound good for you to like it?
I listen mainly to classical music. The SQ of classical recordings is all over the place, not nearly as consistent other types of music. Recording large orchestras is a complicated and difficult endeavor. Smaller ensembles are easier to record. So, if you listen to a great performance of an orchestral (or any) recording but have trouble with the sound will you avoid listening to it?