Source > Preamp > Amp > Speaker
Everything matters, but if you are going to have a heirarchy, your source should be best and things should get gradually worse as the signal heads through the chain. Starting with maximum resolution at the source and then throwing bits away as you go will give better overall sound than any other variation. Ivor T. of Linn has taught this theory for many years. Garbage in > Garbage out.
The loudspeakers are your window into the sound of the electronics and the recordings themselves. No amount of money thrown at a great speaker can recreate or repair a flawed signal from the electronics. Actually, a really transparent speaker will show you more of the limitations of weaker electronics.
Ideally you want to have all things at a comparable level even though it rarely happens in the real world.
Imagine your playing a grown up game of pass it on. You are the captain of a team of 4 players who have to have a very long series of numbers whispered to the first player and then that person must retain as much of the string as possible and whisper it to the 2nd player and so on and so forth. The four players on your team have memory retention abilities which range from poor to superb. As the team captain you must pick the best batting order. What do you do? IF you put the player with the worst memory first, you will have a disaster. That player will lose much of the information right out of the gate while insuring that the better players don't even get the chance to use their superior memory skills to help the team. Your best order would be to have the best player first, 2nd best player next and so on. The same is true of the heirarchy of your stereo system.