More power for moderate listening levels?

This is not an easy question to answer, and it would depend on particulars that are impossible to factor in, such as your own personal preferences and priorities.  My own priority is for the amp to sound good while playing at lower output levels where the vast majority of my listening takes place.  For me personally, I happen to favor the sound of low-powered triode tubes, and even in the solid state realm, the amps I like tend to be lower-powered, such as the gear from First Watt.  This means using fairly efficient speakers (mine are 99 db/w efficient).

Millercarbon raised a very point--lower efficiency speakers cannot really achieve good dynamics no matter how much power is fed because of thermal compression.  As more power is fed to the voice coil, it heats up and that heating increases resistance which reduces the amount of power that can be delivered to the driver.  The less efficient the speaker, the greater the compression.  My difference with his opinion is that I would not be so dismissive of the Harbeths just because they are not efficient; there are plenty of other reasons to like any particular speaker even if it cannot do "really loud."

It might be the case that the higher powered Pass amp will improve the sound, even at lower levels of play, but, I have my doubts about this because caused by some inherent superiority of having much more power in reserve.  This is something you will have to try for yourself to see if it is the case.  A long time ago, I auditioned an am rated to deliver something like 50 watts/channel.  The same company made a 200 watt/channel amp that the dealer said was very similar in design and parts quality, but used more output devices to achieve the higher power capability.  I actually preferred the sound of the lower powered amp in a direct comparison.

I have heard, and liked, the HL5+, and to me, even with lower-powered tube amps, it can play quite loudly.  But, my idea of loud might be different from yours.  In building any speaker, compromises must be made, and I doubt that the HL5+ was designed to have extreme volume level as a strength.  The distortion you thought had been creeping in as the volume got louder might be something in the speaker and not necessarily a problem with the  amplifier you were using.  It would be instructive if you can find a more powerful amp to compare with the one you are using now.

I would personally be more inclined to be looking at amps that sound better at the volume levels I normally listen at, even if it means sacrificing some performance when playing at extremely high volume.  This is how I ended up with three tube amps, the one with the highest power rating being 6.5 watts/channel.