It also depends what are you looking for, and how your hearing is.
For most of us going for higher power is a no-brainer, as most of us have no hearing above 14kHz or so.
However, in case your hearing is still intact, and you are (un)fortunate to hear out to 20kHz, then there's a hard stop. I did this test yesterday - two amps going into a 89dB/Wm true 8R load full range speaker, no crossover, extremely revealing in the high frequencies up to 23kHz. So, easy and direct load and no interference from crossover. I used two amps to drive the speakers.
Amp 1 - 0.6WPC Darling amplifier (SET vacuum tube).
Amp 2 - CJ Premier SA350 Legendary 350WPC solid state power monster. (Which is one of the least fatiguing high power amps you can find.)
To my huge surprise I could not listen to the CJ any louder than I did with the flea-power tube amp! Why? Because yes, it could play MUCH louder, YET the level where it played WITHOUT NOTICEABLE FATIGUE was actually below the level the 0.6W Darling played.
If your hearing is compromised in the HF region, you can push your powerhorse quite a bit more before you notice ANY signs of fatigue. You might be able to use all of it....
Good luck! ;