The maximum power a speaker can handle is an indicator of the full sound spectrum. When the sound is concentrated in only a narrow section of the frequency spectrum, a few watts of electrical power can destroy the driver.
No, the sound spectrum has nothing at all to do with it. It's the voice coil and diaphragm that mostly determine power handling capacity.
Consider the woofer, for example. The ones in my system cover just a few octaves octaves - 20 Hz to 80 Hz yet they can safely handle hundreds of watts.