How much does power amplifier really matter comparing to other hi-fi components?

It all matters from the perspective you need these components in order to create a system. Most of the answers you got are nothing but audiophile nonsense. The amp, preamp, DAC if you go digital are way less important than the speakers and room. If you want a true high fidelity experience just spring for a pair of  Dutch and Dutch 8c and a decent streamer, apply EQ accordingly and room treatments. If you want a passive speaker most of your budget should be speakers and room treatments. As long as you get speakers that are not crazy to drive there are literally dozens of amps and preamps, DACs that will work and don't cost an arm and leg. 

TAS Robert Harley’s book “The Complete Guide To High-End Audio v6” book has an budget example of 36% speakers, 50% amplification+DAC, 9% power conditioner, and 5% interconnects+ cables.  But these percentages are not set in stone- you may get a great deal or discount or may want to pay premium for desired component.  But maybe having a place to start vs no clue is a good thing.

Thanks Kenny,
A Guide book is a good starting point.

I mean Thanks to kennyc .

@djones51 : Agree.