“How much does power amplifier really matter comparing to other hi-fi components?”Lots of variables to consider, particularly speaker sensitivity and listening levels.
Lots and lots.
Generally speaking, you never want to get even close to maximum power, even with short peaks because manufactures usually specify amplifier power when it reaches 1% THD. THD often rises quickly once you get close to maximum power, so keeping 10% below rated power is generally fine and is often close to minimum THD+N.
Provided you are willing to always keep the volume control well below maximum, consider a higher rating than speakers as they are can accept term short peaks without damage.
A good starting point, but always check specifications.