It is the "clipping" of an amplifier which may destroy a speaker. It happens when you overdrive your amplifier (in simplistic terms, this typically means listening to it at very hi volumes).
Regardless of the power of your amplifier, or the power handling capability of your speaker, if you drive your amplifier to clipping levels, then you will definitely have the danger of destroying your speakers.
In your case, since you have not destroyed your speakers, one of these below applies:
- You have not listened to your amplifier at a volume which has yet caused it to clip.
- You may have clip detection circuit in your amplifier.
- You have been lucky and the crossover network in your speaker has managed to protect them.
- You just have been very lucky.