In all my testing, I would probably say the amplifier is the least impactful than any other component. A bad choice in amp can definitely cause your system to sound bad, but as long as you have a generally good choice, the differences between two amps are going to be less then the differences between two choices of preamps/source. The speaker is probably the biggest change, so make sure it's a speaker you generally like the sonic signature of.
The preamp is important, but I would also say your source (DAC/turntable/etc) is also very important. One thing I have found is that if the sound quality / resolution is not there from the source, you are never going to get it back by putting in a super expensive preamp/amp.