How long does it take a decent quality speaker to "wear out"?

Speaker enclosures can wear out as well. Veneers can loosen, hardwoods can warp. Proper, rugged woodworking should see them last decades, however. Foam surrounds deteriorate over time, though newer types (from the last two decades or so) last somewhat longer, depending on their specific implementation. Not only direct sunlight and certain atmospherics can be their enemy over time, but also prodigious excursion. The cone and surround material that has stood the test for the longest time are paper cone variants and cloth surrounds. Paper cones, even older ones, usually have been given a moisture-repellent treatment, certainly pro units that can see use in harsh, outdoor conditions, and cloth surrounds - be they treated or not - last close to forever in fairly stable environments. Indeed, pro drivers are typically built like tanks. Put them at use in domestic environments and they're cruising along without being thermally or mechanically challenged, while sitting in relatively stable indoor humidity and temperature conditions. Rubber surrounds, even modern varieties, can become inflexible and crack, while other (butyl) variants last decades without issues. Rubber surrounds can be treated to remain flexible. Compression drivers run like forever. Some ESL's are very durable, though newer Quad's have a history of depressing failure rate, also being quite sensitive to the environment where they're used. Overall, it would seem the best way to ensure longevity of the mechanical parts in speakers is frequent use over the years, while generally avoiding driving them to their limits.