Do Harbeth speakers really need a 4000 damping factor?

Maybe I’m wrong, but I believe that Dampening factors have very little to do with what brand of speakers you own.

Damping factor is an amplifier measurement.  It is 8 divided by the output impedance of the amp.  Almost all solid state amps have an output impedance so low that the resulting damping factor is WAY in excess of anything meaningful as to the sound; it is largely an irrelevant specification.  The exception is tube amps which tend to have a much higher output impedance, and therefore, a lower damping factor.  There are some amps with output impedance of 2 ohms or so, for a damping factor of 4.  These amps tend to work best with speakers with high and flat impedance curves; with lower impedance speakers, the frequency response will be affected.  Tube amp manufacturers cannot easily reduce output impedance to increase damping factor, anything done for this purpose, such as increasing the turn ratio of the output transformer, will compromise the sound.  That is why speaker matching to some tube amps can be trickier.  

PrimaLuna's EVO 400 power amp has a damping factor of 7.  It's their most powerful pure tube amp ever.

The Crown Studio Reference II has a factor of >20,000.

The answer is no, 240 is plenty.

Your amp is not "lowly".

My 40.3XDs are shared as mains between a T+A integrated and a Yamaha RX3080 AVR. I often stream SXM radio from the Yamaha for casual listening and they sound surprisingly good. 

Prior to the 40's, I did the same with 30.2 XDs - very enjoyable.

The stands matter, as do cables. I've use Ton Trager, but if you can build something similar from hardwood go for it, but get rid of metal frames and glass.

Buy some decent speaker cables, for example from Frank at Signal Cable. They are not expensive - I own the entry level cables and have compared them with my Acoustic Zen Hologram II and the are damn good for the money.

Then, maybe find a used Hegel 590 or similar if budget allows.