Have any of you taken apart a speaker driver to see what's in there? Surrounding and supporting the voice coil, you'll see the "spider". Think of it as the other "driver surround".
Typically, this is made from epoxy-saturated cloth, and pressed into shape. Over time, the spider develops micro-fractures that makes the spider more pliable. One of the reasons you're not recommended to "break in" a speaker with a single tone (sine wave) is you want micro-fractures to occur throughout the spider to support the entire bandwidth of tones the driver will be asked to reproduce.
The spider is far more (but not always more rigid) rigid than the typical foam / rubber surround simply because of the mass of the item - the voice coil - it's supporting.
Yes, yes...and you're "breaking in" the copper in the binding posts as well. That is...if you believe in "speaker break-in".