I definitely think there is something to this. In my experience you need a decent sized woofer to give the right punch and energy above the deepest bass. A pair of bookshelf speakers with a sub can sound almost indistinguishable from large floorstanders on small scale, unamplified music or relatively clean sounding pop. I've never found that configuration satisfying on orchestral music or full-bodied rock, though. It can't pull off the dynamics.
I also think there are some tradeoffs between the ability to play loudly and the ability to sound right at lower volume. Of all of the speakers I've had I've never had a pair that I thought really excelled at both. I'm guessing that in order to play really loudly without distortion drivers need to be built more heavily, with more mass, and that extra mass doesn't respond as well at low volume as lower mass drivers do. Whatever the cause this has been a tradeoff I've noticed consistently.
I also think there are some tradeoffs between the ability to play loudly and the ability to sound right at lower volume. Of all of the speakers I've had I've never had a pair that I thought really excelled at both. I'm guessing that in order to play really loudly without distortion drivers need to be built more heavily, with more mass, and that extra mass doesn't respond as well at low volume as lower mass drivers do. Whatever the cause this has been a tradeoff I've noticed consistently.