enough amplifier power

I am curious as to why so many people think that their amplifiers are powerful enough for their speakers. I use a Yamamoto A-08S--around 1.5 watts output. I use it with a Fostex F-106ESR. The combination is a little ragged at low volumes, but beautifully immediate. Distorts awfully at anything approaching a decent volume. I see people using 20-100 watt amplifiers with medium efficiency loudspeakers. I do not see how this can work any better. If you work out the math, most loudspeakers need 200-500 watts minimum. That is not even taking into account low impedance loudspeakers. Do people not know what distortion sounds like? Or, compression either, for that matter? Please enlighten me.