A flat frequency response is crucial for a natural representation of the music. You don’t want the system to artificially boost or depress certain frequencies, and hence change the sound from what it was. As has been remarked, there are two sides to this.
First there is the speaker. It should have a relatively flat impedance curve without deep dips. There are plenty of speakers that present an easy load (Harbeth are good, but there are enough others), but there are also speakers with a devilish load.
Second there is the load dependency of the amplifier. A good amplifier should not be too troubled by impedance swings. Tube amps have a harder time here, given their design with output transformers. The Prima Luna that Eric referred to, with its absolutely massive frequency swings, is an extreme example of how not to design an amplifier. Compare this to an affrodable Yamaha AS 500, tested here:
http://www.avhub.com.au/product-reviews/hi-fi/yamaha-a-s500-amplifier-review-test-395710 Even under realistic speaker loads it maintained a flat frequency response within 0.05 dB, an amazing performance not matched by many amps costing ten times more.
Taken together, I think speakers that are not easy to drive and amplifiers that are load dependent are just bad engineering, and are to be avoided. If you succumb, you will suffer the dreaded synergy problem and all the audiophile Angst that comes with it.
The second problem is power output. Most good speakers are not very efficient, so you will need lots of power to drive them. If you don’t have enough power, you will suffer clipping distortion on dynamic peaks, with a ’dirty’ distorted and compressed sound as a result. Again, some people like this, but it does not meet the criterion of neutral and life like representation. So how much do you need, not for average levels, but for dynamic peaks? A few years ago, having moved to a larger house and having upgraded from the Quad els 57 speakers to the less efficient Quad 2805 I decided to replace the 2x45 watt Quad 303 by a 2x140 watt Quad 606-2. There was no diference on small scale music, but it sounded much more realistic on the large and dynamic repertoire. I began to realize that I now also played my music rather louder, because the sound remained so clean. Since then, I have concluded that even more power may well further improve the sound. As another anecdote, Harbeth's Alan Shaw recently demonstrated his big M40.1 speakers in the Netherlands. Even he was surprised that on dynamic music the big amplifier that was used was putting out more than 500 watt per channel. Fortunately these days watts are no longer particularly expensive.