I don't know what percentage of high-end speakers are considered 'hard-to-drive' but I think it's a shame one would want to exclude them simply for watts. You might be missing 'your speaker', disqualifying it needlessly.
These days (like computer memory) good sounding watts don't have to break the bank. There are very good A/B and D amps out there for speakers wanting watts.
Like many (most?) others my system is designed around loudspeakers. To me, they are the single most important component in any musical reproduction system, and the most different sounding, when compared to electronics, especially. (I guess you could design a system around a CD transport if you wanted to take it there.)
But if you did buy hard-to-drive, lean-sounding speakers, huge tube amps might break the bank if that's your only out... And personally, most all of the extremely high-efficient designs sound a bit too analytical for me. Warming up the sound with a high distortion little tube amp is humorous (and I've owned tubes as well).