Low power amps and speaker damage?

I've always read that low power amps are more likely to damage speakers than high powered ones (provided they're not overdriven).  This normally in threads where a member asks something like:  Will my 200 watt amp damage my speakers rated at 100 watts?

I've driven several 86db, or thereabouts, efficient speakers with just a few watt tubed amps and have never damaged my speakers.

Was I just lucky or is it I didn't push the speakers into loud enough volume for the low watts to damage the drivers?

Thanks, and stay safe.

Mamoru