Even if you are a true music lover (many aren't) your musical taste buds must be trained just as your taste buds for food must be trained. The first time I heard Led Zeppelin in '69, I didn't care for it. But it touched something to the point that I said, "Lets play it again." I had the same reaction to every other Zep album, even though I am a big Zep fan and John Bonham protege.
That said, I still have not developed the taste for jazz and female vocals. I've tried and it just doesn't move me. So I don't buy anymore even though Elusive Disk sends me emails with their latest audiophile music (jazz & female vocals) and tell me how fabulous it is. It isn't to me. Quite honestly it isn't to 98-99% of the US. But many audiophiles buy this stuff because they think the SHOULD like it. been there, ain't going back.I am a classic rock, blues fan who bought my first album in 1963 (beach Boys Live) I had to train my musical taste buds at first, to like most all the music on an album. I still like most all the music on an album I buy.
The question is are you listening to the right music? The right music is music you like, not necessarily what someone else thinks is posh or "sophisticated".