Interesting conversation. Have restored a few sets of vintage speakers. Always replace the XO caps. most of the time, don't even test them, just replace.
Every time it has been a night and day difference for the better! It always opens the speaker up, more bass, cleaner midrange, highs star to sing.
If you are worried about ESR, try to get caps that don't have a low ESR. Even a cap that might register correct value might have a super high ESR, telling you it needs to be replaced. My last set of speakers a couple caps were in spec, but ESR was about 1ohm.
From my experience, new caps always read on the low side, old caps are either on the high side, or just at value. If you have a 100uf old cap reading at 100uf or over, it's most likely about done.
If you are unsure, just do one speaker, compare to the old. if you don't like it, swap it back.