If you're just getting started, the speakers and room are the places to spend your resources, but at some point it all matters. Once the system reaches a higher level, you need to more careful to match anything new that gets added to the system level or it'll become the weak link....and the weak link is usually matters most.
And the biggest influence on sound quality is...
The quality of the recording itself.
Then the room, the setup, the speakers, and lastly the front end.
I've got recordings that make my system sound horrible, and I've got recordings that make my system sound absolutely wonderful.
None of the gear changes have had that much impact on sound quality.