Speakers are a very important component. I would put the sound of the speakers above all. It is pretty easy to evaluate speakers and just check to see they are not really inefficient. You want speakers you love the sound of, then tweak your electronics to get to most out of them. So, just stay away from planar and <87db or so.
One of the greatest changes I made to my system was going from solid state amps to tubed. While I have less than half the power, no question my system sounds the best it ever has… although to be honest, while the wattage halved, the cost did double… so probably not surprising it sounds so much better.