Why are high efficiency speakers preferred for low volume listening?

I am sure that this is a very basic principle, but as I peruse the speaker section I frequently see high efficiency speakers suggested for those who listen at "low levels." And is this another area that actually is "how easy the speaker is to drive (as related to its nominal impedance)" that is more important than the actual sensitivity number?

And for an example of what I am asking with that last sentence, I seem to remember when I was window shopping for speakers, seeing some Harbeth speakers at TMR with a sensitivity rated below 87 (I think they were rated at 86 or 85) but being referred to as "an easy load to drive." So would that mean that the Harbeth speakers would be good for low volume listening?