I think the reality is a little complicated, but you are right that most of us never exceed 20 Watts listening.
I think it may best be explained that some speakers are a lot harder to drive than you would think, and an amp that can handle them without sagging tends to be bigger, heavier and higher power.
By sagging I mean both reduced overall output but also changes in frequency response. That is, the ideal amplifier would output the same frequency response regardless of speaker, but as we get further and further from ideal the amplifier’s output tends to start to mirror the speaker impedance, even at low listening volumes.
One of the best sources of information about this are the Stereophile amplifier reviews. Take a look at the measurements where they test with a simulated speaker load, and pick a few tube amps vs. a few solid state amps and you'll start to see a pattern.