My bet is that you don't love your system.
I remember doing the same thing—always listening for the good bits and always aware of the bad bits in the reproduction chain.
Eventually, over many tweaks, things clicked. Now I have no issues listening and everything sounds wonderful. Something is amiss and you need to locate it.