Definitely not always. We also can't really say this is true about having more damping factor.
The amp/speaker match up is important as is the quality of the first few watts of power. There are definitely speakers with troublesome impedance curves that like big amps with low output impedance, but also large speakers with big woofers that can benefit from high output impedance. See Nelson Pass's writing for the latter.