How long does it take a decent quality speaker to "wear out"?

It depends on the materials used....some can go on for lifetimes. Foam surrounds rot and will need to be replaced usually within 20-25 years. Some caps will also need to be replaced at 20-30 years, others will go much longer. Ferro fluid will dry out in time and need to be replaced in 20-30 years. With some maintenance, many can live on as good as new.

My personal speakers are pushing 33 years old with components that are over 35 years old. The butyl surrounds and kevlar cones are in great shape, as are all the crossover parts. Solder joints all look good. If anything, the well flexed spiders and surrounds work better than ever. Even the paint and cloth grills still look great.  The only thing I’ve replaced was ferro fluid in the tweeters.

Durability and lifetime depends on specific design.  I have heard speakers with 70 year old drivers that sound terrific.  Paper woofers with pleated fabric surrounds can be quite durable compared to even modern woofers with rubber surrounds.  My midrange compression drivers are about 80 years old. Even speakers with surrounds that go bad can be long-lived if you accept that surrounds have to be replaced every twenty years or so.

i own a set of B&W 801 series 2 with Kevlar cones which were never played very hard never in direct sun light which i purchased way back in 1984 all i have done

is to keep the cabinets oiled so they would not dry out and the dust off the covers with the original brush i received with the speakers, I am using a Krell built

Bass alignment Filter which was designed for that series along with bi-amping and they still sound as good as the day i purchased them.

Still my Dunlavy’s (bought new) show no signs of age but they are well looked after.

Crossovers are still like new, no electrolytics to worry about, no corrosion or bad joints, internal cabling perfect. Never pushed really hard, but always connected to big amplifiers, rubber surrounds talc powdered every 3 years and cabinets always clean and shiny as new. Keep them or not they should perform nicely for years to come. Still if there is any deterioration in sound due to age i cannot tell but i am not going to replace any part just to find out.

Have an old pair of Spectrum 208A made in Toledo,Ohio.  The firm closed in the 90s.  Recently had them re-grilled and re-foamed by a speaker guru outside St. Louis.  

Still have the same cross overs ... still surprise me!