1)the most important part of the system is the room (unless your room is perfect). The first problem encountered is not knowing that this lack of focus, sounstage or whatever, is due to room resonances, etc. There is plenty of information on the web about basic acoustics and DIY projects. With a little chance and some sweat you can fix some problems of your room and dedicate to the other "most important parts". By this I mean that everything counts. Sorry.
A mid-fi system in a perfect room will sound better than a HiEnd system on a poor room (the sentence is not mine).
2)then you have to deal with NOISE. When you think you got rid of noise you still have plenty of it! You wont know it untill you try good cables (there are cheap good quality cables around) or some specialized stuff. There are different kinds of noise. In digital, as important as noise is JITTER. Do your homework. I won't extend on that because I'm not an expert, but my excellent recent experiences with Monarchy DIP de-jitter box, stillpoint ERS sheets and Bybee Quantum purifiers made me a believer. Paraphrasing the sentence in (1) I would say that Mid-fi equipment properly treated for noise/jitter sound closer to Hi-end gear(for a relative small fee). And tweaking is fun.
3) of course speakers are important, but If you buy a great, good and pleasant sounding pair of speakers you may never discover that you have a big problem concerning points 1) and 2). If you are still enjoying it, it's great. The problem is when you start to get tired of the sound and you don't know why. I think if my system/music sounds great with cheap speakers then it's time to concentrate on speakers and upgrade them (or adding more Bybee's). It is difficult to realize wich is the limiting step on your chain. Price is not always truthful.
4)As the sound improuves so does your hearing cappacity/accuracy and you will find two things(appart augmenting the listening pleasure):
a)you will find pleasure listening to other stereos of lesser quality (for a moment).
B)you will be compeled to upgrade. Try Yoga.
Good comments, Brainwaters.