there is a difference between a speaker being sufficiently driven versus the same speaker being driven to show all it can do at higher volumes
alot depends on the type of music, how much dynamic range is needed
smaller wattage amps often do the mids and nuances very well, but will sound compressed and/or flabby/unconvincing in the bass when the music become more dynamic
big power amps might not be as highly tactile at lower volumes but will provide a great sense of ease and naturalness as the music becomes more dynamic and complex
the sweet spot in all this depends on the specific case... speakers, taste of listener, type of music listened to, room, and volume levels