Here's an example that I hope sheds some light on your question. I have gone down the road of low sensitivity/difficult impedance speakers driven with a large amp. My Thiel CS6 speakers have an 86 dB sensitivity and the impedance drops below 3 ohms for a significant part of the frequency spectrum. They also present a difficult phase angle which requires a large amp that can supply high current. My Krell KSA 300S amp outputs 300 watts @ 8 ohms, 600 watts @ 4 ohms, 1200 watts @ 2 ohms, and 2400 watts @ 1 ohm. Very few amplifiers can do this. BTW, this amp weighs 185 lbs and when run loud gets hot enough so that the heat sinks glow in the dark (OK, I'm kidding about the glowing part but it's too hot to touch).
When you hook up an amp to these speakers that is not designed to provide high current into lower impedances the amp acts like a tone control. For the CS6 the lowest impedance is in the upper bass/lower midrange region and the speakers would sound thin in the bass with a typical tube amp. Will the amp work? Sure, it will make sound. You may even like the sound but you are not hearing what the designer intended.
The difference is not subtle. When I had my KSA 300S recapped I tried hooking up my CS6's to my 100 watt Onkyo AV receiver. I literally laughed out loud when I played the first cut. The sound was awful - the bass was anemic, the soundstage collapsed, and the dynamics were compressed - even at low volumes. BTW, this receiver sounds just fine driving a pair of Polk Audio speakers that have a more conventional sensitivity and impedance curve.
Your Harbeths are low sensitivity but present an easy impedance curve which would allow you to experiment with low wattage amplifiers. I suspect these speakers would sound better with more watts but it would be an interesting comparison.
Here's a good video by John DeVore about this subject. He designs high sensitivity speakers with flat impedance curves that are easy to drive. My Krell would be total overkill driving one of DeVore's models. DeVore Rant